Oct 30 21:12:56.670964 ip-10-0-132-23 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:12:57.098846 ip-10-0-132-23 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:12:57.098846 ip-10-0-132-23 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:12:57.098846 ip-10-0-132-23 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:12:57.098846 ip-10-0-132-23 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:12:57.103786 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103713 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.103786 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103787 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103791 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103795 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103798 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103801 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103804 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103807 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103816 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103819 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103822 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103825 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103827 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103830 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103833 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103836 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103839 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103842 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103844 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103847 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103850 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.103854 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103852 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103854 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103857 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103860 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103863 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103866 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103869 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103871 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103874 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103877 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103879 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103882 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103884 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103887 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103889 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103892 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103894 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103897 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103900 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.104313 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103902 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103905 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103907 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103910 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103913 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103915 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103918 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103920 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103922 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103925 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103928 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103930 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103932 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103936 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103938 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103941 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103944 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103947 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103949 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103952 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.104813 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103955 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103957 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103959 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103962 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103965 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103967 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103971 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103973 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103976 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103978 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103981 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103983 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103986 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103989 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103991 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103994 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103996 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.103999 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104001 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104004 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.105285 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104006 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104008 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104011 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104013 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104016 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104018 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104021 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104024 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104027 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104029 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104032 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104034 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104037 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104039 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104042 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104044 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104047 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104051 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104054 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.105766 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104057 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104060 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104062 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104065 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104068 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104071 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104074 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104831 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104839 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104842 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104844 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104848 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104851 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104854 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104857 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104859 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104862 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104865 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104868 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104870 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.106208 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104873 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104876 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104879 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104882 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104884 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104887 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104890 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104892 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104895 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104898 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104900 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104903 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104905 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104908 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104910 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104913 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104916 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104919 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104921 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.106689 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104925 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104927 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104930 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104935 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104939 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104942 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104944 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104947 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104950 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104952 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104956 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104960 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104963 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104966 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104969 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104972 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104975 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104977 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104980 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.107155 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104982 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104985 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104988 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104990 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104993 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104996 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.104998 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105001 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105003 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105006 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105008 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105011 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105013 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105016 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105019 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105021 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105024 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105027 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105031 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105033 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.107636 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105036 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105038 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105041 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105044 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105047 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105049 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105052 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105054 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105057 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105060 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105062 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105065 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105068 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105070 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105073 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105075 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105078 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105080 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105083 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105085 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.108131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105088 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105090 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105092 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105096 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105099 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105101 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105104 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105113 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105116 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105119 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105122 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105125 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105127 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105130 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105133 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105607 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105612 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105615 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105618 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.108624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105621 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105623 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105626 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105629 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105631 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105634 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105637 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105639 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105642 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105645 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105647 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105650 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105652 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105655 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105657 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105660 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105663 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105666 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105669 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.109131 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105671 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105674 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105677 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105680 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105683 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105686 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105688 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105691 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105693 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105696 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105698 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105701 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105717 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105720 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105722 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105725 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105728 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105730 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105733 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.109624 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105737 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105741 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105744 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105747 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105750 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105752 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105755 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105757 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105760 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105763 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105765 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105768 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105772 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105774 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105780 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105783 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105786 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105789 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105792 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105794 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.110139 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105798 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105803 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105806 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105809 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105812 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105814 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105817 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105819 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105822 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105824 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105827 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105830 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105832 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105835 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105837 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105840 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105842 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105845 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105847 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.110673 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105850 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105852 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105855 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105857 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105860 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105862 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105865 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105868 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105871 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105874 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105878 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105881 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105884 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105887 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105890 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105892 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105895 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105897 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105900 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105902 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.111205 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105905 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105908 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105910 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105913 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.105915 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106859 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106869 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106876 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106881 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106885 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106889 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106893 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106897 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106900 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106903 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106907 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106910 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106914 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106917 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106922 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106925 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106928 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106932 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:12:57.111759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106936 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106939 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106942 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106946 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106949 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106953 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106956 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106960 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106963 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106966 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106969 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106973 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106976 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106979 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106983 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106986 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.106997 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107000 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107003 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107006 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107011 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107014 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107017 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107020 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:12:57.112346 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107023 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107027 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107030 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107033 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107039 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107042 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107045 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107048 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107052 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107055 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107058 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107061 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107065 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107068 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107071 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107074 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107077 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107080 2583 flags.go:64] FLAG: --help="false" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107084 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107087 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107090 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107093 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107097 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:12:57.113060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107101 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107104 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107107 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107110 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107112 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107115 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107119 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107121 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107124 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107127 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107130 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107133 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107136 2583 flags.go:64] FLAG: --lock-file="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107140 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107144 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107147 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107152 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107156 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107160 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107163 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107166 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107170 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107173 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107176 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107180 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:12:57.113631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107183 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107187 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107190 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107193 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107196 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107199 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107203 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107206 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107209 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107215 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107218 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107221 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107225 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107228 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107231 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107234 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107236 2583 flags.go:64] FLAG: --port="10250" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107240 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107242 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-009105aa62ae44ccc" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107245 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107250 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107253 2583 flags.go:64] FLAG: --register-node="true" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107256 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107260 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107263 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107266 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:12:57.114252 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107269 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107273 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107276 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107279 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107282 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107285 2583 flags.go:64] FLAG: --runonce="false" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107288 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107291 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107294 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107297 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107300 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107303 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107306 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107309 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107312 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107315 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107318 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107321 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107324 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107327 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107330 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107335 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107338 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107341 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107345 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:12:57.114876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107348 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107352 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107356 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107359 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107362 2583 flags.go:64] FLAG: --v="2" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107367 2583 flags.go:64] FLAG: --version="false" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107372 2583 flags.go:64] FLAG: --vmodule="" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107376 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107380 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107512 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107516 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107520 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107523 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107525 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107528 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107531 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107534 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107536 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107539 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107542 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107544 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.115471 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107547 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107549 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107552 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107555 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107557 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107560 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107562 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107565 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107567 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107570 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107573 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107575 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107579 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107583 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107586 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107589 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107591 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107594 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107596 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107599 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.115996 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107602 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107605 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107607 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107611 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107615 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107618 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107622 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107625 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107628 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107631 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107635 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107638 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107641 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107644 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107647 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107650 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107652 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107655 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.116463 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107657 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107660 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107662 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107665 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107667 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107670 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107673 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107677 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107680 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107682 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107685 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107687 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107690 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107693 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107695 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107698 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107701 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107716 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107720 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107725 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.116940 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107729 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107731 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107734 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107737 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107739 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107742 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107744 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107747 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107750 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107753 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107755 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107757 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107760 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107762 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107765 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107767 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107770 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107772 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107775 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.117484 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107779 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107782 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107784 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107787 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107790 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107793 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107795 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107798 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107800 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107803 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107806 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107809 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107811 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107814 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107817 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107819 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.117951 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107821 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.107848 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107884 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107888 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107892 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107896 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107900 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107903 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107905 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107908 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107911 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107914 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107916 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107919 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107921 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.118312 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107924 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107928 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107931 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107933 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107936 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107938 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107941 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107943 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107946 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107948 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107951 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107954 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107957 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107959 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107962 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107965 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107967 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107970 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107972 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107975 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.118687 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107977 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107980 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107982 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107985 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107988 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107990 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107993 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107996 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.107998 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108001 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108003 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108006 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108009 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108013 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108015 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108018 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108020 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108023 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108025 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.119180 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108027 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108030 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108032 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108035 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108037 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108041 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108043 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108046 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108048 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108051 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108053 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108055 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108058 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108061 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108063 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108066 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108068 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108071 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108073 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108077 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.119630 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108079 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108082 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108085 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108087 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108090 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108093 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108096 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108099 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108101 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108104 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108106 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108109 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108112 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108114 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108117 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108119 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108122 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108124 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108127 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108130 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.120109 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108134 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108137 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108139 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108142 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108145 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108147 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108150 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108152 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108155 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108158 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108160 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108164 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108166 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.108169 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.115973 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.115989 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.115999 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.116004 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116060 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.120577 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116065 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116068 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116073 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116077 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116081 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116083 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116086 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116089 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116091 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116094 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116097 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116100 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116103 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116106 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116108 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116111 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116114 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116116 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116119 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116121 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.121056 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116124 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116126 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116129 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116132 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116134 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116137 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116140 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116143 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116146 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116150 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116152 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116155 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116157 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116160 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116162 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116166 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116170 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116173 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116176 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.121517 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116180 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116183 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116186 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116192 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116195 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116198 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116201 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116204 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116206 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116209 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116211 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116214 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116217 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116219 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116222 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116224 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116227 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116229 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116232 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.122002 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116235 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116238 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116241 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116244 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116247 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116249 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116252 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116254 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116257 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116260 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116262 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116265 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116268 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116270 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116273 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116276 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116278 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116281 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116283 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116286 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.122450 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116289 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116291 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116294 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116296 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116299 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116301 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116304 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116306 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116309 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116311 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116314 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116316 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116319 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116321 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116325 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116327 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116330 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116333 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116335 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116338 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.122934 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116341 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116343 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116346 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116348 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116351 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116353 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116356 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.116383 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116521 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116526 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116529 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116532 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116534 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116537 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116539 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116542 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.123408 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116545 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116547 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116550 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116552 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116555 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116557 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116560 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116562 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116565 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116567 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116571 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116575 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116579 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116583 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116585 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116588 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116590 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116593 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116595 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.123797 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116598 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116600 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116603 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116606 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116608 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116611 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116613 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116616 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116618 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116621 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116623 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116626 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116628 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116631 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116633 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116636 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116638 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116641 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116644 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116646 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.124240 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116649 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116653 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116656 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116659 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116662 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116665 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116668 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116671 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116673 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116676 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116678 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116680 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116683 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116686 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116688 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116691 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116693 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116696 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.124838 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116698 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116701 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116717 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116720 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116722 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116725 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116727 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116730 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116732 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116735 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116738 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116740 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116743 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116746 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116749 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116751 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116754 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116756 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116759 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116761 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.125296 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116764 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116767 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116770 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116772 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116775 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116778 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116781 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116784 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116786 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116789 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116791 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116794 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116797 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116799 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116802 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116804 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116807 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116810 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116812 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116815 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.125863 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116818 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.116841 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116870 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116874 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116877 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116879 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116885 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116887 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116890 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116892 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116895 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116898 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116900 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116903 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116905 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116908 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:12:57.126535 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116911 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116914 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116916 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116919 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116921 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116924 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116927 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116929 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116932 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116935 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116938 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116940 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116943 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116948 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116952 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116955 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116957 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116960 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116963 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:12:57.126985 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116966 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116968 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116971 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116976 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116979 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116982 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116984 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116987 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116990 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116992 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116995 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.116997 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117000 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117003 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117006 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117008 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117011 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117013 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117016 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:12:57.127592 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117018 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117022 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117025 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117027 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117030 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117033 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117035 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117038 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117040 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117045 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117047 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117050 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117052 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117055 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117058 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117060 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117064 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117067 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117069 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117072 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:12:57.128057 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117075 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117077 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117080 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117083 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117085 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117088 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117090 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117094 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117096 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117099 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117101 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117104 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117107 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117109 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117112 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117114 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117117 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117119 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117122 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117124 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:12:57.128633 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117127 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117131 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117133 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117136 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117138 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117141 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117144 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117146 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117150 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117152 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117155 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117160 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117163 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:57.117166 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.117214 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.121428 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.122508 2583 server.go:1014] "Starting client certificate rotation" Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.122599 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:12:57.129270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.126977 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:12:57.129799 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.128581 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:12:57.129799 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.129740 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:12:57.144505 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.144484 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:12:57.148362 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.148348 2583 log.go:25] "Validated CRI v1 image API" Oct 30 21:12:57.149218 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.149195 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:12:57.153783 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.153759 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d60a68f4-b920-4a42-90cc-dab0b95bc8e5:/dev/nvme0n1p3 f17da894-7b78-43b9-9bd6-f6945ecf3e44:/dev/nvme0n1p4] Oct 30 21:12:57.153856 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.153782 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:12:57.160223 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.160111 2583 manager.go:217] Machine: {Timestamp:2026-10-30 21:12:57.158606788 +0000 UTC m=+0.377414609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106027 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258f8a64012d40e3125b5487fb8835 SystemUUID:ec258f8a-6401-2d40-e312-5b5487fb8835 BootID:31d47b79-e27d-453f-a0ed-b1ef58f03b1c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:db:02:f2:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:db:02:f2:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:94:78:2b:f8:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:12:57.160223 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.160221 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:12:57.160333 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.160299 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:12:57.163029 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163003 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:12:57.163163 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163032 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-23.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:12:57.163210 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163165 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:12:57.163210 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163174 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:12:57.163210 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163209 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:12:57.163288 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.163227 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:12:57.164580 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.164570 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:12:57.169189 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.169177 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:12:57.169311 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.169302 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:12:57.170181 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170171 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:12:57.170217 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170185 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:12:57.170217 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170196 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:12:57.170217 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170205 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:12:57.170217 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170213 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:12:57.170559 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.170545 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:12:57.173212 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.173193 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:12:57.180738 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.180721 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:12:57.181676 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181665 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:12:57.181747 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181695 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:12:57.181858 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181849 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:12:57.181890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181864 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:12:57.181890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181871 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:12:57.181890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181877 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:12:57.181890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181883 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:12:57.181890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.181888 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:12:57.182545 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182537 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:12:57.182574 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182546 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:12:57.182574 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182554 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:12:57.182574 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182559 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:12:57.182574 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182568 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:12:57.182689 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.182577 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:12:57.183180 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.183171 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:12:57.183180 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.183180 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:12:57.186919 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.186908 2583 server.go:1263] "Started kubelet" Oct 30 21:12:57.187080 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.187025 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:12:57.187080 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.187037 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:12:57.187187 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.187109 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:12:57.187698 ip-10-0-132-23 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:12:57.188483 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.188458 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:12:57.188860 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.188846 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:12:57.193780 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.193759 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:12:57.195658 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.195643 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:12:57.195730 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.195680 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:12:57.195793 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.195757 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.196512 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196495 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:12:57.196512 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196497 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:12:57.196646 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196527 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:12:57.196646 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196585 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:12:57.196646 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196594 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:12:57.196800 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196722 2583 factory.go:153] Registering CRI-O factory Oct 30 21:12:57.196800 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196735 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 21:12:57.196878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196800 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:12:57.196878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196810 2583 factory.go:55] Registering systemd factory Oct 30 21:12:57.196878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196815 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:12:57.196878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196826 2583 factory.go:103] Registering Raw factory Oct 30 21:12:57.196878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.196834 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 21:12:57.197981 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.197970 2583 manager.go:319] Starting recovery of all containers Oct 30 21:12:57.208650 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.208635 2583 manager.go:324] Recovery completed Oct 30 21:12:57.214821 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.214805 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.216415 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.216384 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.216501 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.216430 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.216501 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.216445 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.216981 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.216966 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.217078 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.216964 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:12:57.217078 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.217060 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:12:57.217189 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.217084 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:12:57.217659 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.217642 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.217748 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.217672 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.217748 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.217683 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.219279 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.219264 2583 policy_none.go:50] "Start" Oct 30 21:12:57.219335 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.219282 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:12:57.219335 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.219292 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:12:57.221186 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.221175 2583 policy_none.go:44] "Start" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.254278 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.255130 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.256231 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.256262 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.256285 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.256324 2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.257434 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.257467 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.257479 2583 server.go:87] "Starting device plugin registration server" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.257688 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.257699 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.257955 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.258020 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.258027 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.258392 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:12:57.265357 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.258436 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.326032 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.325991 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-23.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:12:57.326032 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.326000 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-23.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:12:57.327109 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.325966 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-23.us-west-2.compute.internal.18e36b0c4c43d3f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-23.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-23.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:12:57.186882545 +0000 UTC m=+0.405690369,LastTimestamp:2026-10-30 21:12:57.186882545 +0000 UTC m=+0.405690369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-23.us-west-2.compute.internal,}" Oct 30 21:12:57.357257 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.357200 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal"] Oct 30 21:12:57.357257 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.357249 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.357891 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.357878 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.358035 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358018 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.358136 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358052 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.358136 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358067 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.358538 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358525 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.358607 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358549 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.358607 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358560 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.358607 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.358579 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.360241 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360227 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.360390 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.360479 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360406 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.360853 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360839 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.360917 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360859 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.360917 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.360868 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.361046 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.361028 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.361133 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.361053 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.361133 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.361063 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.363118 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.363105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.363169 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.363127 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:57.363759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.363746 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:57.363804 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.363768 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:57.363804 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.363777 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:57.382634 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.382601 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-23.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:12:57.382741 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.382644 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:12:57.382741 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.382586 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-23.us-west-2.compute.internal.18e36b0c4e067353 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-23.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-23.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-23.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:12:57.216414547 +0000 UTC m=+0.435222374,LastTimestamp:2026-10-30 21:12:57.216414547 +0000 UTC m=+0.435222374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-23.us-west-2.compute.internal,}" Oct 30 21:12:57.382741 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.382684 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:12:57.382902 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.382743 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 21:12:57.390094 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.390078 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.392327 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.392312 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.397539 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.397525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.397589 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.397549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.397589 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.397567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.418658 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.418639 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:12:57.418720 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.418692 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-26ml9" Oct 30 21:12:57.485356 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.485338 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.485429 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.485358 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-23.us-west-2.compute.internal\": node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.497759 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.497845 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.497845 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.497845 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.497845 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.497976 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.497847 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.593080 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.593060 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.693868 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.693837 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.694960 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.694945 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.698171 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:57.698152 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:57.794860 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.794726 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.895235 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.895207 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:57.995462 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:57.995431 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.096320 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.096302 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.122345 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.122318 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:12:58.122716 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.122445 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa9979ce62f3e4f9a86916a6635549a1-220db26bc9a348ee.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-23.us-west-2.compute.internal.18e36b0c4e06cc2c\": read tcp 10.0.132.23:58632->16.59.72.207:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-23.us-west-2.compute.internal.18e36b0c4e06cc2c default 4830 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-23.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-23.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-23.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:12:57 +0000 UTC,LastTimestamp:2026-10-30 21:12:57.358553937 +0000 UTC m=+0.577361758,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-23.us-west-2.compute.internal,}" Oct 30 21:12:58.196755 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.196726 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.298439 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.298376 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.370968 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.370947 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:12:58.371808 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.371791 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:12:58.371876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.371821 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:12:58.371876 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.371832 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:12:58.399303 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.399282 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.424929 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.424892 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:07:58 +0000 UTC" deadline="2028-04-17 12:38:05.078905468 +0000 UTC" Oct 30 21:12:58.424929 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.424927 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12831h25m6.653981995s" Oct 30 21:12:58.480881 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.480863 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:12:58.500328 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:58.500308 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:12:58.535284 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.535260 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:12:58.590363 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.590303 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:12:58.595871 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.595849 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:58.665956 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.665936 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:12:58.667010 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.666998 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:12:58.722979 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.722961 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:12:58.731394 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.731375 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:12:58.921726 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:58.921667 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2617ac5c29b7705758059c1bd8aae3.slice/crio-ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38 WatchSource:0}: Error finding container ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38: Status 404 returned error can't find the container with id ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38 Oct 30 21:12:58.922197 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:58.922173 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d614578e9abb3302e5026d743d87fe.slice/crio-f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc WatchSource:0}: Error finding container f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc: Status 404 returned error can't find the container with id f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc Oct 30 21:12:58.927317 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:58.927293 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:12:59.173652 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.173605 2583 apiserver.go:52] "Watching apiserver" Oct 30 21:12:59.197060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.197030 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:12:59.261948 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.261897 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"76d614578e9abb3302e5026d743d87fe","Type":"ContainerStarted","Data":"f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc"} Oct 30 21:12:59.263073 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.263050 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerStarted","Data":"ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38"} Oct 30 21:12:59.288216 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.288189 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:12:59.288552 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.288529 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal","openshift-image-registry/node-ca-8nvnx","openshift-multus/multus-additional-cni-plugins-ng4gs","openshift-multus/multus-v2lxx","openshift-network-operator/iptables-alerter-sw2mt","openshift-ovn-kubernetes/ovnkube-node-gdshh","kube-system/global-pull-secret-syncer-47nbz","kube-system/konnectivity-agent-c5d68","kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-kh6x7","openshift-dns/node-resolver-5xv6n","openshift-network-diagnostics/network-check-target-2qw92","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z","openshift-cluster-node-tuning-operator/tuned-dlx72"] Oct 30 21:12:59.292470 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.292450 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.292565 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.292514 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:12:59.294750 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.294733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.297110 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.297092 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.299325 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.299305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.301647 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.301628 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.304154 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.304138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.306951 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.306930 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.307113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.306962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.307113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.306994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.307113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.307113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307293 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307341 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.307392 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtgb\" (UniqueName: \"kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307612 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4k6\" (UniqueName: \"kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kn4c\" (UniqueName: \"kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307777 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307857 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307889 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.307991 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.308027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.308878 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.308055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.309327 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.309307 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.311815 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.311795 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.314385 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.314369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.314474 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.314420 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:12:59.316601 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.316583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.318931 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.318913 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:12:59.319019 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.318962 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:12:59.352773 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.352752 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.352773 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.352787 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:12:59.352773 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.352771 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.353012 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.352946 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6zwr\"" Oct 30 21:12:59.354171 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.354156 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.354625 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.354605 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:12:59.354895 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.354877 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:12:59.354951 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.354880 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.381666 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381641 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:12:59.381666 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381643 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:12:59.381829 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381702 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzjzq\"" Oct 30 21:12:59.381829 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381724 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.381829 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381782 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:12:59.381829 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381801 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.381950 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381874 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.381950 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.381917 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:12:59.382036 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.382024 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvf2h\"" Oct 30 21:12:59.382079 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.382033 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-79sqh\"" Oct 30 21:12:59.382079 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.382053 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.382150 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.382078 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqpcg\"" Oct 30 21:12:59.387113 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.387098 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:12:59.397607 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.397584 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:12:59.400350 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.400335 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:12:59.400598 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.400584 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.400651 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.400611 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.401542 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.401526 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.401738 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.401707 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.401819 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.401747 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:12:59.401819 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.401808 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:12:59.401894 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.401841 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:12:59.408071 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408054 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:12:59.408071 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408064 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph6zc\"" Oct 30 21:12:59.408256 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408241 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.408296 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408271 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.408343 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.408379 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.408379 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408377 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.408482 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.408482 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.408482 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408494 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408497 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408507 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408537 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7lwhk" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408538 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408586 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h52wm\"" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408586 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408620 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4v9xp\"" Oct 30 21:12:59.408636 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408646 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408657 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408742 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dpfbq\"" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408855 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kn4c\" (UniqueName: \"kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.408995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409344 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409196 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409222 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409048 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409242 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409277 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpv4q\" (UniqueName: \"kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409290 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409299 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409356 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409380 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.409904 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409410 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409421 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409424 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409448 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409529 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mcn\" (UniqueName: \"kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxb8f\" (UniqueName: \"kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409666 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6r5\" (UniqueName: \"kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.410492 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409739 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xr8r\" (UniqueName: \"kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtgb\" (UniqueName: \"kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409940 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.409987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410085 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4k6\" (UniqueName: \"kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410218 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.411227 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410459 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsdm\" (UniqueName: \"kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410535 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410737 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.411767 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410771 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410786 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.410816 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.410888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:59.910867961 +0000 UTC m=+3.129675797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.410998 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.412264 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.411016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.474043 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.473992 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:12:59.477749 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.477724 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4k6\" (UniqueName: \"kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.477843 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.477810 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kn4c\" (UniqueName: \"kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.477843 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.477810 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtgb\" (UniqueName: \"kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.512072 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512044 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.512142 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512142 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512142 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512142 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512201 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512233 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpv4q\" (UniqueName: \"kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512273 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512199 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512348 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512398 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mcn\" (UniqueName: \"kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxb8f\" (UniqueName: \"kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512497 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512560 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6r5\" (UniqueName: \"kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.512711 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512654 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr8r\" (UniqueName: \"kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512687 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512821 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512840 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsdm\" (UniqueName: \"kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512944 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512987 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513012 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513078 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.513583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.512893 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513138 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513155 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513212 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513221 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513263 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513293 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513280 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513326 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513289 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513339 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.514464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513385 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513392 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513421 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513546 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513646 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513761 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513843 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.513846 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.513904 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.513956 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:00.013939027 +0000 UTC m=+3.232746845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.514135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.515242 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.515345 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.515557 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.515982 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.515630 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.577622 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.577598 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:12:59.577622 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.577623 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:12:59.577795 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.577636 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:12:59.577795 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.577754 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:00.077735768 +0000 UTC m=+3.296543600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:12:59.579677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.579641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6r5\" (UniqueName: \"kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.579946 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.579929 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr8r\" (UniqueName: \"kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:12:59.580562 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.580543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mcn\" (UniqueName: \"kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.580648 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.580589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxb8f\" (UniqueName: \"kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.580648 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.580615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsdm\" (UniqueName: \"kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.580749 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.580617 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpv4q\" (UniqueName: \"kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.610836 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.610819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:12:59.620507 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.620484 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd123e3_d34f_411d_a77f_d147f5e3368e.slice/crio-3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36 WatchSource:0}: Error finding container 3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36: Status 404 returned error can't find the container with id 3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36 Oct 30 21:12:59.624869 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.624853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2lxx" Oct 30 21:12:59.628872 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.628854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:12:59.631069 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.631046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:12:59.633669 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.633633 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8bf47b_05dc_4987_a7e9_7fa682c8ab60.slice/crio-0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d WatchSource:0}: Error finding container 0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d: Status 404 returned error can't find the container with id 0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d Oct 30 21:12:59.642585 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.642565 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c729fe8_4a69_477a_a247_1e750d51c96a.slice/crio-2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d WatchSource:0}: Error finding container 2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d: Status 404 returned error can't find the container with id 2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d Oct 30 21:12:59.643880 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.643861 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08289af5_6136_4988_9455_9f2eef24caef.slice/crio-0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930 WatchSource:0}: Error finding container 0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930: Status 404 returned error can't find the container with id 0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930 Oct 30 21:12:59.647138 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.647122 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:12:59.649370 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.649346 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:12:59.654267 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.654242 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:12:59.656118 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.656099 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:12:59.656355 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.656338 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a2ed00_d2f6_45cd_9085_7e93ce8031e1.slice/crio-a6e49644031cc7ee30fbac385d80197c3ee17e7f5bbf2137e13ce447a62df08c WatchSource:0}: Error finding container a6e49644031cc7ee30fbac385d80197c3ee17e7f5bbf2137e13ce447a62df08c: Status 404 returned error can't find the container with id a6e49644031cc7ee30fbac385d80197c3ee17e7f5bbf2137e13ce447a62df08c Oct 30 21:12:59.657613 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.657595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:12:59.659532 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.659463 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da5bf9f_3892_4b54_a337_880b1e19faf8.slice/crio-f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706 WatchSource:0}: Error finding container f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706: Status 404 returned error can't find the container with id f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706 Oct 30 21:12:59.664344 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.664321 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a69c4_438f_4679_9255_d0cae7ffc19f.slice/crio-d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1 WatchSource:0}: Error finding container d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1: Status 404 returned error can't find the container with id d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1 Oct 30 21:12:59.667600 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.667584 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cd5ea6_5cf5_46b7_a778_7a2921fbd837.slice/crio-b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79 WatchSource:0}: Error finding container b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79: Status 404 returned error can't find the container with id b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79 Oct 30 21:12:59.670466 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:12:59.670432 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22b5a4e_5e79_4380_b6d2_2d7eb3e5b04d.slice/crio-f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5 WatchSource:0}: Error finding container f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5: Status 404 returned error can't find the container with id f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5 Oct 30 21:12:59.917691 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:12:59.917657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:12:59.917829 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.917814 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:12:59.917900 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:12:59.917870 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:00.917854354 +0000 UTC m=+4.136662184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:00.018270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.018241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:00.018393 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.018375 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:00.018467 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.018437 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:01.018417639 +0000 UTC m=+4.237225451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:00.118957 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.118916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:00.119102 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.119079 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:00.119102 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.119106 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:00.119273 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.119120 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:00.119273 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.119174 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:01.119157362 +0000 UTC m=+4.337965194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:00.266014 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.265943 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1"} Oct 30 21:13:00.267631 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.267600 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sw2mt" event={"ID":"9c729fe8-4a69-477a-a247-1e750d51c96a","Type":"ContainerStarted","Data":"2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d"} Oct 30 21:13:00.269541 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.269514 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2lxx" event={"ID":"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60","Type":"ContainerStarted","Data":"0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d"} Oct 30 21:13:00.271089 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.271065 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xv6n" event={"ID":"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d","Type":"ContainerStarted","Data":"f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5"} Oct 30 21:13:00.272403 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.272378 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5d68" event={"ID":"2da5bf9f-3892-4b54-a337-880b1e19faf8","Type":"ContainerStarted","Data":"f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706"} Oct 30 21:13:00.274041 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.274014 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlx72" event={"ID":"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1","Type":"ContainerStarted","Data":"a6e49644031cc7ee30fbac385d80197c3ee17e7f5bbf2137e13ce447a62df08c"} Oct 30 21:13:00.276455 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.276432 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerStarted","Data":"0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930"} Oct 30 21:13:00.277899 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.277870 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" event={"ID":"dfd123e3-d34f-411d-a77f-d147f5e3368e","Type":"ContainerStarted","Data":"3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36"} Oct 30 21:13:00.279122 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.279095 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8nvnx" event={"ID":"71cd5ea6-5cf5-46b7-a778-7a2921fbd837","Type":"ContainerStarted","Data":"b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79"} Oct 30 21:13:00.411915 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.411872 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:07:59 +0000 UTC" deadline="2028-06-06 03:49:15.18615657 +0000 UTC" Oct 30 21:13:00.411915 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.411909 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14022h36m14.774251609s" Oct 30 21:13:00.924717 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:00.924675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:00.924932 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.924882 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:00.925005 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:00.924961 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:02.924935423 +0000 UTC m=+6.143743235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:01.025368 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.025334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:01.025566 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.025550 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:01.025634 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.025614 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:03.025596007 +0000 UTC m=+6.244403834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:01.126446 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.126410 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:01.136072 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.136047 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:01.136221 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.136080 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:01.136221 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.136104 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:01.136221 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.136176 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:03.136147995 +0000 UTC m=+6.354955805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:01.259931 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.259841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:01.260093 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.259972 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:01.260093 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.260063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:01.260215 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.260135 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:01.260215 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.260193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:01.260326 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:01.260273 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:01.412171 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.412103 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:07:59 +0000 UTC" deadline="2028-04-26 15:09:56.236546048 +0000 UTC" Oct 30 21:13:01.412171 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:01.412168 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13049h56m54.824381857s" Oct 30 21:13:02.943272 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:02.942862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:02.943272 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:02.943102 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:02.943272 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:02.943175 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:06.943154024 +0000 UTC m=+10.161961837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:03.043764 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.043522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:03.043892 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.043703 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:03.043960 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.043893 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:07.043870594 +0000 UTC m=+10.262678422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:03.145142 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.145113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:03.145345 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.145320 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:03.145345 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.145346 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:03.145486 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.145359 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:03.145486 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.145417 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:07.145395244 +0000 UTC m=+10.364203057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:03.259620 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.259504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:03.259790 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.259649 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:03.259790 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.259722 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:03.259928 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.259831 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:03.259928 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.259885 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:03.260036 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:03.259957 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:03.311362 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.310491 2583 generic.go:356] "Generic (PLEG): container finished" podID="0f2617ac5c29b7705758059c1bd8aae3" containerID="d1e8179fc42a69d86edd4ce0c7b4117dbc23779dbb0eb1feb15ed01673026fb4" exitCode=0 Oct 30 21:13:03.311362 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.310587 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerDied","Data":"d1e8179fc42a69d86edd4ce0c7b4117dbc23779dbb0eb1feb15ed01673026fb4"} Oct 30 21:13:03.328873 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:03.328844 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"76d614578e9abb3302e5026d743d87fe","Type":"ContainerStarted","Data":"89a3057ec0be59840fdc1e4618d421977c8c1b225f54533c2684a7d5707a51fa"} Oct 30 21:13:04.336858 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:04.336816 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="7a7337b91ed5b4f84ec18ef7c52b3ae05ff15b5cb52793ad1f30c7d6b8e86e36" exitCode=0 Oct 30 21:13:04.337381 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:04.336925 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"7a7337b91ed5b4f84ec18ef7c52b3ae05ff15b5cb52793ad1f30c7d6b8e86e36"} Oct 30 21:13:04.351478 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:04.351027 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerStarted","Data":"45441e5e0180ce5d37ae2a951bc902e9768c9d0c2e62b3ee3e31c091126ecb3d"} Oct 30 21:13:04.475003 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:04.474939 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" podStartSLOduration=5.474919369 podStartE2EDuration="5.474919369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:03.569971487 +0000 UTC m=+6.788779342" watchObservedRunningTime="2026-10-30 21:13:04.474919369 +0000 UTC m=+7.693727205" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.257866 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:05.258002 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.258087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:05.258240 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.258400 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:05.258816 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:05.258493 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:05.359989 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.359858 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5d68" event={"ID":"2da5bf9f-3892-4b54-a337-880b1e19faf8","Type":"ContainerStarted","Data":"038b546e375ef904106a501bd5bb971c56e9e020661ba729284228879e034411"} Oct 30 21:13:05.489429 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.489364 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c5d68" podStartSLOduration=2.521795558 podStartE2EDuration="7.489341626s" totalImagesPullingTime="4.967546068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:05.48893598 +0000 UTC m=+8.707743817" watchObservedRunningTime="2026-10-30 21:13:05.489341626 +0000 UTC m=+8.708149462" Oct 30 21:13:05.489915 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:05.489876 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" podStartSLOduration=6.489864668 podStartE2EDuration="6.489864668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:04.601708701 +0000 UTC m=+7.820516538" watchObservedRunningTime="2026-10-30 21:13:05.489864668 +0000 UTC m=+8.708672503" Oct 30 21:13:06.365867 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:06.365829 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8nvnx" event={"ID":"71cd5ea6-5cf5-46b7-a778-7a2921fbd837","Type":"ContainerStarted","Data":"d89a6ec8288077515628dcd667df2b0011897d6852ba741427bd1a80b3d145ec"} Oct 30 21:13:06.645373 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:06.645331 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:06.646191 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:06.646163 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:06.780388 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:06.780335 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8nvnx" podStartSLOduration=2.879207705 podStartE2EDuration="8.780313877s" totalImagesPullingTime="5.901106172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:06.494380612 +0000 UTC m=+9.713188447" watchObservedRunningTime="2026-10-30 21:13:06.780313877 +0000 UTC m=+9.999121715" Oct 30 21:13:06.987484 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:06.987400 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:06.987660 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:06.987569 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:06.987660 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:06.987631 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:14.987612364 +0000 UTC m=+18.206420200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:07.088447 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.088067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:07.088447 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.088236 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:07.088447 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.088293 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:15.088276297 +0000 UTC m=+18.307084127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:07.189616 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.189247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:07.189616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.189475 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:07.189616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.189504 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:07.189616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.189519 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:07.189616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.189575 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:15.189557146 +0000 UTC m=+18.408364990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.256864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.257012 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.257077 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.257143 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.257395 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:07.257616 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:07.257492 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:07.370956 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:07.370431 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" event={"ID":"dfd123e3-d34f-411d-a77f-d147f5e3368e","Type":"ContainerStarted","Data":"a9209da615c500c55c41f72baef95266df4cb5224bb5c3a8a6eb3c58c8ea0b19"} Oct 30 21:13:08.372340 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:08.372312 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:13:08.984907 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:08.984882 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:13:09.257854 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.257828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:09.257854 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.257842 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:09.258152 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:09.257961 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:09.258152 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:09.258066 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:09.258152 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.258144 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:09.258379 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:09.258213 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:09.263901 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.263229 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:13:08.984897034Z","UUID":"89285693-c4bd-4a57-bf44-a79dac506a54","Handler":null,"Name":"","Endpoint":""} Oct 30 21:13:09.264968 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.264949 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:13:09.265066 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.264980 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:13:09.376947 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.376914 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2lxx" event={"ID":"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60","Type":"ContainerStarted","Data":"a7af767bf891fc74e7fc92c62628d72ffcf27b447dd3061782c7848fb1318e9e"} Oct 30 21:13:09.378957 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.378930 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xv6n" event={"ID":"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d","Type":"ContainerStarted","Data":"a583055370f1de6a457acfaeb8bacb70c219784a4b5bf7c825803afb8b96e956"} Oct 30 21:13:09.381077 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.381049 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerStarted","Data":"2e97187cac90d51261f9e54adcaa594d8ed59ab18adf8b3499507c553525b344"} Oct 30 21:13:09.383655 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.383627 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" event={"ID":"dfd123e3-d34f-411d-a77f-d147f5e3368e","Type":"ContainerStarted","Data":"8a0dac539fec2510a98af3186852246b2de8a2266b8b81fcaef33939f5e60901"} Oct 30 21:13:09.619912 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.619855 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-v2lxx" podStartSLOduration=2.90395308 podStartE2EDuration="11.619840178s" totalImagesPullingTime="8.715887098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:09.49982938 +0000 UTC m=+12.718637213" watchObservedRunningTime="2026-10-30 21:13:09.619840178 +0000 UTC m=+12.838648008" Oct 30 21:13:09.620334 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:09.620306 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5xv6n" podStartSLOduration=2.696191002 podStartE2EDuration="11.62030104s" totalImagesPullingTime="8.924110038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:09.619796883 +0000 UTC m=+12.838604715" watchObservedRunningTime="2026-10-30 21:13:09.62030104 +0000 UTC m=+12.839108870" Oct 30 21:13:10.388119 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:10.388083 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="2e97187cac90d51261f9e54adcaa594d8ed59ab18adf8b3499507c553525b344" exitCode=0 Oct 30 21:13:10.388544 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:10.388418 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"2e97187cac90d51261f9e54adcaa594d8ed59ab18adf8b3499507c553525b344"} Oct 30 21:13:10.391557 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:10.391531 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sw2mt" event={"ID":"9c729fe8-4a69-477a-a247-1e750d51c96a","Type":"ContainerStarted","Data":"f3afffd48d04f0a3a8ce9bba5fb7aa2e6de41c3ab43e815aec54a0f2c43d88c4"} Oct 30 21:13:11.257157 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.256987 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:11.257308 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.256973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:11.257308 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:11.257244 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:11.257308 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.256999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:11.257438 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:11.257338 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:11.257438 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:11.257387 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:11.394477 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.394448 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlx72" event={"ID":"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1","Type":"ContainerStarted","Data":"daf62fe58cacf9b1bdfd0ac2b9690f492d3e3ca6e2a904c676e2c2fcad71697f"} Oct 30 21:13:11.397057 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.397026 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" event={"ID":"dfd123e3-d34f-411d-a77f-d147f5e3368e","Type":"ContainerStarted","Data":"a85ee6be3b751aa00444c43e3b5a78073e9090379cd1d843698905cb30d81e5b"} Oct 30 21:13:11.514601 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.514554 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sw2mt" podStartSLOduration=4.289356244 podStartE2EDuration="13.514539437s" totalImagesPullingTime="9.225183193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:10.643889307 +0000 UTC m=+13.862697139" watchObservedRunningTime="2026-10-30 21:13:11.514539437 +0000 UTC m=+14.733347260" Oct 30 21:13:11.515056 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.515028 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlx72" podStartSLOduration=2.104314151 podStartE2EDuration="13.515018865s" totalImagesPullingTime="11.410704714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:11.514502211 +0000 UTC m=+14.733310066" watchObservedRunningTime="2026-10-30 21:13:11.515018865 +0000 UTC m=+14.733826705" Oct 30 21:13:11.635424 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:11.635378 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" podStartSLOduration=3.030844622 podStartE2EDuration="13.635365859s" totalImagesPullingTime="10.604521237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:11.634838023 +0000 UTC m=+14.853645855" watchObservedRunningTime="2026-10-30 21:13:11.635365859 +0000 UTC m=+14.854173729" Oct 30 21:13:12.401084 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:12.401050 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="659efc90a9b01f24b4070576c59c8afdb408db372be1c7e50589968e1ae3520d" exitCode=0 Oct 30 21:13:12.401774 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:12.401098 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"659efc90a9b01f24b4070576c59c8afdb408db372be1c7e50589968e1ae3520d"} Oct 30 21:13:13.257514 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:13.257482 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:13.257678 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:13.257482 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:13.257678 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:13.257594 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:13.257678 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:13.257620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:13.257868 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:13.257701 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:13.257868 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:13.257833 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:14.406786 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:14.406750 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="208c1e7de732f5a499108b9cc2698fca331bf3d099200683b807142a90e44572" exitCode=0 Oct 30 21:13:14.407374 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:14.406823 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"208c1e7de732f5a499108b9cc2698fca331bf3d099200683b807142a90e44572"} Oct 30 21:13:15.058224 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.058191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:15.058383 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.058300 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:15.058383 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.058349 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret podName:d7cd2ff9-7aeb-4ebf-9605-a213958c1153 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.058332868 +0000 UTC m=+34.277140691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret") pod "global-pull-secret-syncer-47nbz" (UID: "d7cd2ff9-7aeb-4ebf-9605-a213958c1153") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:15.158508 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.158476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:15.158640 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.158569 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:15.158640 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.158612 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.158600132 +0000 UTC m=+34.377407941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:15.257079 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.256896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:15.257236 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.256920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:15.257236 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.257159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:15.257341 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.257276 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:15.257341 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.256952 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:15.257430 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.257406 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:15.258850 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.258832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:15.258965 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.258930 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:15.258965 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.258944 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:15.258965 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.258954 2583 projected.go:196] Error preparing data for projected volume kube-api-access-5mzxp for pod openshift-network-diagnostics/network-check-target-2qw92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:15.259106 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:15.258996 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp podName:06a00eb6-ceaa-49ee-8a17-01ba1009767e nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.258979731 +0000 UTC m=+34.477787540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mzxp" (UniqueName: "kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp") pod "network-check-target-2qw92" (UID: "06a00eb6-ceaa-49ee-8a17-01ba1009767e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:15.410308 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:15.410278 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"d8ef692d79d5be5939e4a2ff7e0bf8f731515cc12e0c470681babdc1dc059f5f"} Oct 30 21:13:16.416080 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.415576 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerStarted","Data":"715b2545af3aab8c1d219d04cffa6b0ba088ad83f730d7849a8c1b0c79292caa"} Oct 30 21:13:16.418548 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.418528 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:16.418961 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.418938 2583 generic.go:356] "Generic (PLEG): container finished" podID="3b7a69c4-438f-4679-9255-d0cae7ffc19f" containerID="e00fc900a6f206e61fac07a2e08801ce79dd5f6ddf603bd9ded4d046c99f2354" exitCode=1 Oct 30 21:13:16.419029 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.418977 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"63fefa2dc8bb1f50f3ade615a2f6b43779646192f8cdb5584b954515a0caa456"} Oct 30 21:13:16.419067 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.419035 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"302bf35d8f56f871648d98a3a51aede8cc6affc1badad3ba55aa973dfb56d09f"} Oct 30 21:13:16.419096 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.419079 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"7a641aa6d711231314d08ca96b884164e0d41d5f76d7f7f5ea859c6fbbba7361"} Oct 30 21:13:16.419132 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.419117 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"23e48ce71b9707df70caa22b8c1654c74c3be0f809c28ad1eacad17acf01e9d2"} Oct 30 21:13:16.419165 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:16.419154 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerDied","Data":"e00fc900a6f206e61fac07a2e08801ce79dd5f6ddf603bd9ded4d046c99f2354"} Oct 30 21:13:17.257717 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:17.257685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:17.257890 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:17.257835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:17.257890 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:17.257853 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:17.258034 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:17.257971 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:17.258034 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:17.258009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:17.258131 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:17.258086 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:17.423464 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:17.423433 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="715b2545af3aab8c1d219d04cffa6b0ba088ad83f730d7849a8c1b0c79292caa" exitCode=0 Oct 30 21:13:17.423857 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:17.423494 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"715b2545af3aab8c1d219d04cffa6b0ba088ad83f730d7849a8c1b0c79292caa"} Oct 30 21:13:18.428854 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:18.428820 2583 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="cd8f45ea957ed0faf9e11de5939230c3a29f851d87e5914f475f1baedb3aea2a" exitCode=0 Oct 30 21:13:18.429445 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:18.428901 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerDied","Data":"cd8f45ea957ed0faf9e11de5939230c3a29f851d87e5914f475f1baedb3aea2a"} Oct 30 21:13:18.431992 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:18.431978 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:18.432372 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:18.432354 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"18144bfed73a8eb4403ab5bfdef471850cb8ab1df4ac47f6b003c4dc914a634d"} Oct 30 21:13:19.256924 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:19.256891 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:19.256924 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:19.256919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:19.257150 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:19.256942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:19.257150 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:19.257020 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:19.257150 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:19.257093 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:19.257259 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:19.257171 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:19.437201 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:19.437160 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" event={"ID":"08289af5-6136-4988-9455-9f2eef24caef","Type":"ContainerStarted","Data":"28bfe14b3f176cb21e5f19232266d0a0bc9a7e5db67f2ec95a84a6b208f07be2"} Oct 30 21:13:19.611310 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:19.611272 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ng4gs" podStartSLOduration=8.634697237 podStartE2EDuration="21.611260096s" totalImagesPullingTime="12.976562859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:19.611014362 +0000 UTC m=+22.829822193" watchObservedRunningTime="2026-10-30 21:13:19.611260096 +0000 UTC m=+22.830067927" Oct 30 21:13:20.442349 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:20.442169 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:20.442928 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:20.442670 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"5fe82df4b51153eca43158895c34b532e3e8e7c09e3672d551ad1ed7624cc69b"} Oct 30 21:13:20.443282 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:20.443266 2583 scope.go:122] "RemoveContainer" containerID="e00fc900a6f206e61fac07a2e08801ce79dd5f6ddf603bd9ded4d046c99f2354" Oct 30 21:13:21.046216 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.046173 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:21.181121 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.181090 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:21.181270 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.181228 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:13:21.181580 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.181562 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:21.256848 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.256824 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:21.256848 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.256837 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:21.256994 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.256825 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:21.256994 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:21.256904 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:21.257148 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:21.257030 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:21.257148 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:21.257117 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:21.448336 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.448312 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:21.448754 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.448656 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" event={"ID":"3b7a69c4-438f-4679-9255-d0cae7ffc19f","Type":"ContainerStarted","Data":"976bc890ad5ad8f76afd0cb00701d9fda5fb4b3273b043c7eb56818509eba763"} Oct 30 21:13:21.448998 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.448971 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:21.449153 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.449133 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:21.463611 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.463587 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:21.463829 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.463744 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:21.655585 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:21.655538 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" podStartSLOduration=8.048982897 podStartE2EDuration="23.65552573s" totalImagesPullingTime="15.606542833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:21.628913583 +0000 UTC m=+24.847721413" watchObservedRunningTime="2026-10-30 21:13:21.65552573 +0000 UTC m=+24.874333561" Oct 30 21:13:22.536268 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.536239 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kh6x7"] Oct 30 21:13:22.536618 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.536347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:22.536618 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:22.536436 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:22.538520 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.538497 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qw92"] Oct 30 21:13:22.538626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.538575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:22.538675 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:22.538636 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:22.541220 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.541202 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-47nbz"] Oct 30 21:13:22.541292 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:22.541281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:22.541375 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:22.541361 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:24.257313 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:24.257257 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:24.257762 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:24.257326 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:24.257762 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:24.257407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:24.257762 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:24.257409 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:24.257762 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:24.257519 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:24.257762 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:24.257592 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:26.256800 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:26.256766 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:26.256800 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:26.256797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:26.257277 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:26.256863 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:26.257277 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:26.256883 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47nbz" podUID="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" Oct 30 21:13:26.257277 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:26.256933 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qw92" podUID="06a00eb6-ceaa-49ee-8a17-01ba1009767e" Oct 30 21:13:26.257277 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:26.257011 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kh6x7" podUID="dce12b75-7554-47a5-8214-f40f414060d7" Oct 30 21:13:28.099660 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.099501 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeReady" Oct 30 21:13:28.099958 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.099752 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:13:28.217800 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.217744 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m4s2v"] Oct 30 21:13:28.221312 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.221295 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jz9n2"] Oct 30 21:13:28.221437 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.221412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:28.226175 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.226149 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7qqx"] Oct 30 21:13:28.226317 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.226288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:28.230119 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.229973 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:13:28.230172 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.230137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:28.233944 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.233927 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2q86m"] Oct 30 21:13:28.234081 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.234059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.237630 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.237615 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s"] Oct 30 21:13:28.237766 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.237748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.240888 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.240868 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp"] Oct 30 21:13:28.241004 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.240967 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.244163 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.244147 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl"] Oct 30 21:13:28.244248 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.244234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.248567 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.248548 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq"] Oct 30 21:13:28.248946 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.248930 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:28.252119 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.252102 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsh9q"] Oct 30 21:13:28.252221 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.252207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.255563 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.255542 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b999d9c47-t55lr"] Oct 30 21:13:28.255789 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.255772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.258831 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.258814 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:13:28.258921 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.258907 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:28.258971 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.258937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:28.259262 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.259183 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:28.259354 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.259306 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.262378 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.262018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.272321 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.272294 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-crfvx"] Oct 30 21:13:28.276460 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.276431 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5"] Oct 30 21:13:28.276681 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.276652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.279321 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.279304 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rk98"] Oct 30 21:13:28.279413 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.279400 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.282137 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.282118 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd"] Oct 30 21:13:28.282248 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.282235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.285107 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.285090 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:28.286723 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.286692 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m4s2v"] Oct 30 21:13:28.292774 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.292758 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.302235 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.302216 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2cbsq"] Oct 30 21:13:28.303286 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.303258 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:13:28.303623 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.303604 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:13:28.303725 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.303637 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lrfkn\"" Oct 30 21:13:28.303725 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.303661 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:13:28.303725 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.303698 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:13:28.305101 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.305084 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jz9n2"] Oct 30 21:13:28.305175 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.305164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.312130 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.312110 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s"] Oct 30 21:13:28.317090 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.317073 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:13:28.325506 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.325487 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsh9q"] Oct 30 21:13:28.326438 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.326419 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp"] Oct 30 21:13:28.354997 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.354972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksvg\" (UniqueName: \"kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.355096 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flq22\" (UniqueName: \"kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.355096 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662r6\" (UniqueName: \"kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.355096 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhph\" (UniqueName: \"kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph\") pod \"volume-data-source-validator-58c9f5f46d-6f4jl\" (UID: \"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355192 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w28n\" (UniqueName: \"kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n\") pod \"kas-connection-checker-685895c7c9-m4s2v\" (UID: \"f9771a55-0c02-4b30-9fe7-1850bd414cde\") " pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.355996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.355540 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356093 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356293 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmgt\" (UniqueName: \"kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356317 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356364 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356508 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbwr\" (UniqueName: \"kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356651 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.357739 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356735 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxs2f\" (UniqueName: \"kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356854 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcfk\" (UniqueName: \"kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.356997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwdb\" (UniqueName: \"kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.358240 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsz6j\" (UniqueName: \"kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j\") pod \"kas-connection-checker-685895c7c9-jz9n2\" (UID: \"323fa83f-8f63-4a65-9837-d31e6f6de11d\") " pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:28.358780 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.358780 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.358780 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2k4d\" (UniqueName: \"kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d\") pod \"kas-connection-checker-685895c7c9-c7qqx\" (UID: \"88f63d46-40a6-46c0-b359-d58509ccba89\") " pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:28.358780 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.357642 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.360870 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.360852 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b999d9c47-t55lr"] Oct 30 21:13:28.361520 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.361503 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5"] Oct 30 21:13:28.362056 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.362045 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-crfvx"] Oct 30 21:13:28.362747 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.362725 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:13:28.364487 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.364469 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2q86m"] Oct 30 21:13:28.369747 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.369728 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7qqx"] Oct 30 21:13:28.373842 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.373823 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl"] Oct 30 21:13:28.387162 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.387132 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388163 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388140 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:13:28.388475 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388210 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388475 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388243 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388475 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388316 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388323 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388397 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388404 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388408 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388412 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388626 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388524 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.388860 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388656 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.388860 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388745 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:13:28.388860 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388754 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.388860 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.388791 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.410852 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.410817 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:13:28.411099 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.411079 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:13:28.411099 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.411094 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:13:28.411099 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.411085 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.411416 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.411384 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.411416 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.411391 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.414418 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414401 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.414418 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414404 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:13:28.414580 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414464 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.414642 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414608 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.414699 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414648 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.414764 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414701 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.414764 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414736 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:28.414764 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.414738 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.419930 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.419913 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:28.437460 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437441 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:13:28.437542 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437449 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:13:28.437588 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437569 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:13:28.437634 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437600 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:13:28.437680 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437653 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:13:28.437680 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437664 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:13:28.437680 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437674 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:13:28.437996 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437982 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:13:28.438041 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.437994 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:13:28.438041 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.438031 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:13:28.438103 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.438041 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:13:28.439421 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.439405 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:13:28.439491 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.439456 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q4s27\"" Oct 30 21:13:28.439491 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.439481 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fn5j\"" Oct 30 21:13:28.439579 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.439493 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwnd5\"" Oct 30 21:13:28.439579 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.439545 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w5d85\"" Oct 30 21:13:28.440109 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.440092 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:13:28.440170 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.440158 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lm9pr\"" Oct 30 21:13:28.440416 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.440401 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq"] Oct 30 21:13:28.441018 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.441003 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2cbsq"] Oct 30 21:13:28.458230 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbwr\" (UniqueName: \"kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.458292 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.458292 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.458292 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.458430 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.458476 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs2f\" (UniqueName: \"kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.458517 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcfk\" (UniqueName: \"kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.458565 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.458565 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.458729 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.458782 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.458838 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.458838 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.458838 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwdb\" (UniqueName: \"kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz6j\" (UniqueName: \"kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j\") pod \"kas-connection-checker-685895c7c9-jz9n2\" (UID: \"323fa83f-8f63-4a65-9837-d31e6f6de11d\") " pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:28.458980 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.458987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k4d\" (UniqueName: \"kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d\") pod \"kas-connection-checker-685895c7c9-c7qqx\" (UID: \"88f63d46-40a6-46c0-b359-d58509ccba89\") " pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459026 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459084 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2blx\" (UniqueName: \"kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459119 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksvg\" (UniqueName: \"kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flq22\" (UniqueName: \"kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459204 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.459260 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662r6\" (UniqueName: \"kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459303 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhph\" (UniqueName: \"kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph\") pod \"volume-data-source-validator-58c9f5f46d-6f4jl\" (UID: \"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459335 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459218 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459433 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.959413712 +0000 UTC m=+32.178221536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459446 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w28n\" (UniqueName: \"kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n\") pod \"kas-connection-checker-685895c7c9-m4s2v\" (UID: \"f9771a55-0c02-4b30-9fe7-1850bd414cde\") " pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459507 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459531 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459544 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459603 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.459787 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459667 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.95965025 +0000 UTC m=+32.178458073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459720 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459772 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.95975879 +0000 UTC m=+32.178566624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.459947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.459968 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.959951508 +0000 UTC m=+32.178759335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460195 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvrw\" (UniqueName: \"kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw\") pod \"network-check-source-6f78d59f99-gbtvd\" (UID: \"f2becdfd-f3fd-4a40-8278-566d27fc1ebb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460381 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.460511 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460647 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460657 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.460731 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.460775 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.960761529 +0000 UTC m=+32.179569340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.460815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.460861 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.460977 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.960962657 +0000 UTC m=+32.179770484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.461112 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmgt\" (UniqueName: \"kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.461677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.461317 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.462059 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.462035 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.463049 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.462719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.463879 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.463855 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:28.463973 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.463941 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.963918938 +0000 UTC m=+32.182726767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:28.464303 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464279 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.464424 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.464297 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:28.964274337 +0000 UTC m=+32.183082168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:28.464529 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464504 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.464683 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464655 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.464886 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464857 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.465010 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464914 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.465010 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.464931 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.465815 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.465781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.466285 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.466256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.466647 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.466618 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.466799 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.466679 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.467526 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.467501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.474596 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.474541 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:13:28.474596 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.474553 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:13:28.475642 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.475625 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:13:28.476175 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.476156 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd"] Oct 30 21:13:28.480316 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.480298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.480477 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.480455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.480583 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.480568 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.482504 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.482486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.489130 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.489113 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rk98"] Oct 30 21:13:28.523471 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.523450 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.524093 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.524069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcfk\" (UniqueName: \"kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.527349 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.527330 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksvg\" (UniqueName: \"kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.530448 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.530430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbwr\" (UniqueName: \"kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.530663 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.530643 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.531308 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.531289 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flq22\" (UniqueName: \"kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.531434 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.531415 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.536060 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.536037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662r6\" (UniqueName: \"kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.536779 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.536730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs2f\" (UniqueName: \"kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.537722 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.537689 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhph\" (UniqueName: \"kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph\") pod \"volume-data-source-validator-58c9f5f46d-6f4jl\" (UID: \"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:28.538347 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.538318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w28n\" (UniqueName: \"kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n\") pod \"kas-connection-checker-685895c7c9-m4s2v\" (UID: \"f9771a55-0c02-4b30-9fe7-1850bd414cde\") " pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:28.539465 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.539449 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz6j\" (UniqueName: \"kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j\") pod \"kas-connection-checker-685895c7c9-jz9n2\" (UID: \"323fa83f-8f63-4a65-9837-d31e6f6de11d\") " pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:28.542589 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.542560 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:28.543324 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.543305 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwdb\" (UniqueName: \"kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.543404 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.543357 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k4d\" (UniqueName: \"kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d\") pod \"kas-connection-checker-685895c7c9-c7qqx\" (UID: \"88f63d46-40a6-46c0-b359-d58509ccba89\") " pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:28.543404 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.543394 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.543536 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.543522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmgt\" (UniqueName: \"kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.547919 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.547900 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:28.561312 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.561290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:28.562603 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.562586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2blx\" (UniqueName: \"kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.562677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.562614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.562677 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.562669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvrw\" (UniqueName: \"kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw\") pod \"network-check-source-6f78d59f99-gbtvd\" (UID: \"f2becdfd-f3fd-4a40-8278-566d27fc1ebb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:28.564889 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.564866 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.567602 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.567583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:28.578102 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.578080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:28.584620 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.584445 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:28.633446 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.633420 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2blx\" (UniqueName: \"kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.635796 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.635774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvrw\" (UniqueName: \"kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw\") pod \"network-check-source-6f78d59f99-gbtvd\" (UID: \"f2becdfd-f3fd-4a40-8278-566d27fc1ebb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:28.637747 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.637489 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:28.656159 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.655965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:28.663670 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.663504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:28.836170 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.836134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:28.936684 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:28.936657 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f63d46_40a6_46c0_b359_d58509ccba89.slice/crio-1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5 WatchSource:0}: Error finding container 1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5: Status 404 returned error can't find the container with id 1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5 Oct 30 21:13:28.937432 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.937408 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7qqx"] Oct 30 21:13:28.939020 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:28.938986 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95e0daf_f36a_4951_8ead_9cf4ef55a928.slice/crio-5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa WatchSource:0}: Error finding container 5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa: Status 404 returned error can't find the container with id 5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa Oct 30 21:13:28.940576 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.940482 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2q86m"] Oct 30 21:13:28.942507 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:28.942485 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323fa83f_8f63_4a65_9837_d31e6f6de11d.slice/crio-59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67 WatchSource:0}: Error finding container 59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67: Status 404 returned error can't find the container with id 59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67 Oct 30 21:13:28.943102 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.943087 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jz9n2"] Oct 30 21:13:28.965832 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.965803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:28.965934 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.965890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.965934 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.965922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:28.966037 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.965948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:28.966037 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.965988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:28.966195 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.966067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:28.966195 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.966102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:28.966195 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.966146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:28.966326 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966273 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:28.966326 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966324 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.966306946 +0000 UTC m=+33.185114770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:28.966477 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966388 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:28.966477 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966425 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.96641365 +0000 UTC m=+33.185221463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:28.966612 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966494 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.96648362 +0000 UTC m=+33.185291443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:28.966612 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966552 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:28.966612 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966586 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.966574068 +0000 UTC m=+33.185381892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:28.966792 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966654 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:28.966792 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966685 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.966674946 +0000 UTC m=+33.185482758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:28.966889 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966849 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:28.966889 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.966876746 +0000 UTC m=+33.185684569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:28.966965 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966894 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:28.966965 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966911 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:28.967055 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966967 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.966951314 +0000 UTC m=+33.185759123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:28.967055 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966849 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:28.967055 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.966989 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:28.967055 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:28.967020 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:29.967009886 +0000 UTC m=+33.185817701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:28.978610 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.978582 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:28.989175 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.989118 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl"] Oct 30 21:13:28.992820 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.991146 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s"] Oct 30 21:13:28.993011 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:28.992861 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aa7aba_cd3c_4caf_8fec_df3b5aa043fa.slice/crio-ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28 WatchSource:0}: Error finding container ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28: Status 404 returned error can't find the container with id ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28 Oct 30 21:13:28.994035 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:28.993966 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp"] Oct 30 21:13:28.996581 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:28.996502 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf904ba1_72fc_41f3_82d7_959374abe111.slice/crio-5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6 WatchSource:0}: Error finding container 5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6: Status 404 returned error can't find the container with id 5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6 Oct 30 21:13:29.049427 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:29.049393 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5cba4a_8119_475d_a365_f7977405ea6e.slice/crio-9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9 WatchSource:0}: Error finding container 9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9: Status 404 returned error can't find the container with id 9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9 Oct 30 21:13:29.049687 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.049664 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-crfvx"] Oct 30 21:13:29.108939 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.103503 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2cbsq"] Oct 30 21:13:29.112578 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:29.112551 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274417e7_ffd3_4de2_a249_a15e27177905.slice/crio-32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db WatchSource:0}: Error finding container 32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db: Status 404 returned error can't find the container with id 32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db Oct 30 21:13:29.200164 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:29.200137 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2becdfd_f3fd_4a40_8278_566d27fc1ebb.slice/crio-9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f WatchSource:0}: Error finding container 9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f: Status 404 returned error can't find the container with id 9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f Oct 30 21:13:29.202018 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.201994 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd"] Oct 30 21:13:29.223050 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:29.223028 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9771a55_0c02_4b30_9fe7_1850bd414cde.slice/crio-c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97 WatchSource:0}: Error finding container c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97: Status 404 returned error can't find the container with id c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97 Oct 30 21:13:29.224754 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.224736 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m4s2v"] Oct 30 21:13:29.262485 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.262420 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:29.466819 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.466786 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerStarted","Data":"9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9"} Oct 30 21:13:29.468313 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.468284 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" event={"ID":"323fa83f-8f63-4a65-9837-d31e6f6de11d","Type":"ContainerStarted","Data":"cdba93642143a4cd4860c678a20a9bfc0e131d55dc41343b5b0af51201297ed0"} Oct 30 21:13:29.468426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.468340 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" event={"ID":"323fa83f-8f63-4a65-9837-d31e6f6de11d","Type":"ContainerStarted","Data":"59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67"} Oct 30 21:13:29.469442 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.469413 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" event={"ID":"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa","Type":"ContainerStarted","Data":"ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28"} Oct 30 21:13:29.470595 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.470573 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" event={"ID":"ec41e273-2c37-467e-b55d-cef56c254d57","Type":"ContainerStarted","Data":"431dc86ba2f6dd72c2dc27b50ba3b404cb6f4a0fc1333cded0785fb4c4585ea8"} Oct 30 21:13:29.471804 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.471779 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2q86m" event={"ID":"e95e0daf-f36a-4951-8ead-9cf4ef55a928","Type":"ContainerStarted","Data":"5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa"} Oct 30 21:13:29.473236 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.473216 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" event={"ID":"88f63d46-40a6-46c0-b359-d58509ccba89","Type":"ContainerStarted","Data":"228599bb958fb5c7691618b7adc527bed4733a828b2dc1fb21496d70b5a7830e"} Oct 30 21:13:29.473307 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.473255 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" event={"ID":"88f63d46-40a6-46c0-b359-d58509ccba89","Type":"ContainerStarted","Data":"1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5"} Oct 30 21:13:29.474643 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.474615 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" event={"ID":"f9771a55-0c02-4b30-9fe7-1850bd414cde","Type":"ContainerStarted","Data":"fb403fd6e7e3f8d4031cc3cd217a1fb5fc978b48c08a4eaebcda359c5339e53b"} Oct 30 21:13:29.474743 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.474650 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" event={"ID":"f9771a55-0c02-4b30-9fe7-1850bd414cde","Type":"ContainerStarted","Data":"c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97"} Oct 30 21:13:29.475753 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.475731 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2cbsq" event={"ID":"274417e7-ffd3-4de2-a249-a15e27177905","Type":"ContainerStarted","Data":"32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db"} Oct 30 21:13:29.476808 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.476790 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" event={"ID":"df904ba1-72fc-41f3-82d7-959374abe111","Type":"ContainerStarted","Data":"5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6"} Oct 30 21:13:29.477797 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.477776 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" event={"ID":"f2becdfd-f3fd-4a40-8278-566d27fc1ebb","Type":"ContainerStarted","Data":"9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f"} Oct 30 21:13:29.974400 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:29.974400 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:29.974577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:29.974577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:29.974577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:29.974577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:29.974577 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974525 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974587 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974569354 +0000 UTC m=+35.193377169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974597 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974630 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974585 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974603 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974602 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974595707 +0000 UTC m=+35.193403515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:29.974756 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:29.974751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974786 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974799 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974813 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.97479561 +0000 UTC m=+35.193603428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974837 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974826769 +0000 UTC m=+35.193634581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974849 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974860 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974851 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974845941 +0000 UTC m=+35.193653751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974890 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974881639 +0000 UTC m=+35.193689448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974901 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974894716 +0000 UTC m=+35.193702526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:29.975006 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:29.974915 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:31.974910829 +0000 UTC m=+35.193718638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:30.451589 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:30.451488 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" podStartSLOduration=387.451468183 podStartE2EDuration="6m27.451468183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:30.450296304 +0000 UTC m=+33.669104136" watchObservedRunningTime="2026-10-30 21:13:30.451468183 +0000 UTC m=+33.670276010" Oct 30 21:13:30.571525 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:30.571467 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" podStartSLOduration=387.571446037 podStartE2EDuration="6m27.571446037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:30.570656504 +0000 UTC m=+33.789464364" watchObservedRunningTime="2026-10-30 21:13:30.571446037 +0000 UTC m=+33.790253870" Oct 30 21:13:30.688916 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:30.687021 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" podStartSLOduration=387.687004251 podStartE2EDuration="6m27.687004251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:30.68673059 +0000 UTC m=+33.905538422" watchObservedRunningTime="2026-10-30 21:13:30.687004251 +0000 UTC m=+33.905812083" Oct 30 21:13:31.087329 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.087279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:31.090455 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.090411 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:31.188613 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.188582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:31.188800 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.188748 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:31.188869 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.188806 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:14:03.188793165 +0000 UTC m=+66.407600975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : secret "metrics-daemon-secret" not found Oct 30 21:13:31.290286 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.290254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:31.294062 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.294008 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:31.302862 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.302786 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:31.316336 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.316135 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:31.774539 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.774444 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-47nbz"] Oct 30 21:13:31.824028 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.823998 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qw92"] Oct 30 21:13:31.826307 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:31.826279 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a00eb6_ceaa_49ee_8a17_01ba1009767e.slice/crio-b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95 WatchSource:0}: Error finding container b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95: Status 404 returned error can't find the container with id b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95 Oct 30 21:13:31.998637 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:31.998820 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:31.998820 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:31.998820 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:31.998820 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:31.999024 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:31.999024 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998876 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:31.999024 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:31.998948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:31.999166 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999088 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:31.999166 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999105 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:31.999166 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999161 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999142081 +0000 UTC m=+39.217949911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:31.999646 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999623 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:31.999646 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999648 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:31.999825 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999692 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999678119 +0000 UTC m=+39.218485929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:31.999825 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999765 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:31.999825 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999795 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999786193 +0000 UTC m=+39.218594002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999840 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999853 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999862 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999882 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999872544 +0000 UTC m=+39.218680353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999885 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999905 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999888452 +0000 UTC m=+39.218696284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999925 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999914246 +0000 UTC m=+39.218722074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999943 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.999932549 +0000 UTC m=+39.218740363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:31.999977 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:31.999962 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.9999524 +0000 UTC m=+39.218760215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:32.492909 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:32.492819 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qw92" event={"ID":"06a00eb6-ceaa-49ee-8a17-01ba1009767e","Type":"ContainerStarted","Data":"b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95"} Oct 30 21:13:32.497274 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:32.496572 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" event={"ID":"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa","Type":"ContainerStarted","Data":"6d803ce2fd26296d8ae87ccd7d6016ccdc87c0e2a57d743505f79833f6479bb1"} Oct 30 21:13:32.498744 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:32.498660 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-47nbz" event={"ID":"d7cd2ff9-7aeb-4ebf-9605-a213958c1153","Type":"ContainerStarted","Data":"8f6fc9fcf1c9d763d0ebb6bc751d120f84f3780df2753ffa426f8707eed19c7a"} Oct 30 21:13:32.779803 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:32.779700 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:33.009920 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:33.009857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:34.513927 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:34.513879 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2q86m" event={"ID":"e95e0daf-f36a-4951-8ead-9cf4ef55a928","Type":"ContainerStarted","Data":"d7b13dfe53cdc2d30339b681bdcf43b10217cf0ba5c13f12102ad410bc5d0443"} Oct 30 21:13:34.636810 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:34.636691 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" podStartSLOduration=346.260744429 podStartE2EDuration="5m48.636674513s" totalImagesPullingTime="2.375930084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:32.616159738 +0000 UTC m=+35.834967571" watchObservedRunningTime="2026-10-30 21:13:34.636674513 +0000 UTC m=+37.855482345" Oct 30 21:13:35.329425 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.329331 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-2q86m" podStartSLOduration=344.571579393 podStartE2EDuration="5m49.329311646s" totalImagesPullingTime="4.757732253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:34.636242778 +0000 UTC m=+37.855050610" watchObservedRunningTime="2026-10-30 21:13:35.329311646 +0000 UTC m=+38.548119477" Oct 30 21:13:35.329997 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.329979 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bn5sg"] Oct 30 21:13:35.352223 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.351505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.433156 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.433123 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bn5sg"] Oct 30 21:13:35.440902 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.440579 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.440902 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.440621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.440902 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.440673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n259\" (UniqueName: \"kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.440902 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.440727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.440902 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.440755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.486261 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.486229 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:13:35.486941 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.486834 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:13:35.487287 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.487249 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9xqh\"" Oct 30 21:13:35.519992 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.519645 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" event={"ID":"ec41e273-2c37-467e-b55d-cef56c254d57","Type":"ContainerStarted","Data":"4eb41b005ec5b46fe67cfd723beaad7d37a804282f3992864b3b742c832a562e"} Oct 30 21:13:35.522601 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.522566 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" event={"ID":"df904ba1-72fc-41f3-82d7-959374abe111","Type":"ContainerStarted","Data":"7566aa62e5f73fcdee0a2d9e618d9dbab649275949ec6fdf712efaf45b4e5b4a"} Oct 30 21:13:35.526670 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.525579 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/0.log" Oct 30 21:13:35.526670 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.525633 2583 generic.go:356] "Generic (PLEG): container finished" podID="de5cba4a-8119-475d-a365-f7977405ea6e" containerID="3eedb0091293ddeb8247b3585d9997291c617d4860358ba6e755c5cc59c054c5" exitCode=255 Oct 30 21:13:35.526670 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.526343 2583 scope.go:122] "RemoveContainer" containerID="3eedb0091293ddeb8247b3585d9997291c617d4860358ba6e755c5cc59c054c5" Oct 30 21:13:35.527131 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.526910 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerDied","Data":"3eedb0091293ddeb8247b3585d9997291c617d4860358ba6e755c5cc59c054c5"} Oct 30 21:13:35.541765 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.541740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.541972 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.541779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.541972 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.541827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n259\" (UniqueName: \"kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.541972 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.541865 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.541972 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.541900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.542107 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:35.542052 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:35.542107 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:35.542103 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:36.042087643 +0000 UTC m=+39.260895458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:35.542372 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:35.542354 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:35.542495 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:35.542407 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:36.042392845 +0000 UTC m=+39.261200668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:35.542745 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.542680 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.542874 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.542779 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.614381 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.614331 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n259\" (UniqueName: \"kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:35.619144 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.619053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:35.638388 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.638343 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" podStartSLOduration=344.097665069 podStartE2EDuration="5m49.638328959s" totalImagesPullingTime="5.54066389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:35.637043799 +0000 UTC m=+38.855851626" watchObservedRunningTime="2026-10-30 21:13:35.638328959 +0000 UTC m=+38.857136790" Oct 30 21:13:35.756832 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.756390 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" podStartSLOduration=343.798085033 podStartE2EDuration="5m49.756372071s" totalImagesPullingTime="5.958287038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:35.755385842 +0000 UTC m=+38.974193674" watchObservedRunningTime="2026-10-30 21:13:35.756372071 +0000 UTC m=+38.975179904" Oct 30 21:13:35.813450 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:35.813424 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.009075 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.009010 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.049033 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.048995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049063 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049104 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049113 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049185 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:37.049168443 +0000 UTC m=+40.267976252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:36.049209 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049227 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049246 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049250 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049295 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049279393 +0000 UTC m=+47.268087207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049329 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049352 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049345771 +0000 UTC m=+47.268153580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.049375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049386 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:36.049426 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049410 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:37.049403123 +0000 UTC m=+40.268210932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049441 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049458 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049451282 +0000 UTC m=+47.268259090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049469 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049463591 +0000 UTC m=+47.268271400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049247 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049479 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049496 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049490364 +0000 UTC m=+47.268298173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049506 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049501145 +0000 UTC m=+47.268308954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049521 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049527 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049545 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049539175 +0000 UTC m=+47.268346984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049549 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:36.049930 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.049568 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.049563069 +0000 UTC m=+47.268370877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:36.201389 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.201315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.530801 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.530761 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2cbsq" event={"ID":"274417e7-ffd3-4de2-a249-a15e27177905","Type":"ContainerStarted","Data":"50476a398dfbdfc19c2eed1ea412ac2e1aaa80a838f08fa49feac8690f52d17a"} Oct 30 21:13:36.532565 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.532544 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:36.533164 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.533137 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/0.log" Oct 30 21:13:36.533284 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.533184 2583 generic.go:356] "Generic (PLEG): container finished" podID="de5cba4a-8119-475d-a365-f7977405ea6e" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" exitCode=255 Oct 30 21:13:36.533284 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.533270 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerDied","Data":"8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a"} Oct 30 21:13:36.533387 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.533325 2583 scope.go:122] "RemoveContainer" containerID="3eedb0091293ddeb8247b3585d9997291c617d4860358ba6e755c5cc59c054c5" Oct 30 21:13:36.533500 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.533479 2583 scope.go:122] "RemoveContainer" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" Oct 30 21:13:36.533742 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:36.533696 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:36.648721 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.648663 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2cbsq" podStartSLOduration=2.067914912 podStartE2EDuration="8.648649272s" totalImagesPullingTime="6.58073436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:36.647760418 +0000 UTC m=+39.866568252" watchObservedRunningTime="2026-10-30 21:13:36.648649272 +0000 UTC m=+39.867457103" Oct 30 21:13:36.787807 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.787679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.967577 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.967547 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd"] Oct 30 21:13:36.977078 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.977045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.987329 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:36.987304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:37.027299 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.027270 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd"] Oct 30 21:13:37.059691 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.059618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:37.059833 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.059810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:37.059966 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:37.059947 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:37.060086 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:37.059969 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:37.060086 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:37.060021 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:39.060003252 +0000 UTC m=+42.278811061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:37.060086 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:37.060044 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:39.060030807 +0000 UTC m=+42.278838619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:37.118016 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.117991 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:37.118422 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.118383 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9zv66\"" Oct 30 21:13:37.118422 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.118388 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:13:37.160852 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.160826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz72g\" (UniqueName: \"kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g\") pod \"migrator-66df5d496b-tbvbd\" (UID: \"d0bd86df-4448-46c9-aac0-2e8496b22533\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:37.162534 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.162458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.261447 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.261416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz72g\" (UniqueName: \"kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g\") pod \"migrator-66df5d496b-tbvbd\" (UID: \"d0bd86df-4448-46c9-aac0-2e8496b22533\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:37.323845 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.323779 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz72g\" (UniqueName: \"kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g\") pod \"migrator-66df5d496b-tbvbd\" (UID: \"d0bd86df-4448-46c9-aac0-2e8496b22533\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:37.355702 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.355538 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.538219 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.538183 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" event={"ID":"f2becdfd-f3fd-4a40-8278-566d27fc1ebb","Type":"ContainerStarted","Data":"a5c545ab260a073d5461dd6476e87c8c32134ed782fabedf5bf48db5586663cd"} Oct 30 21:13:37.540001 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.539973 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:37.540334 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.540312 2583 scope.go:122] "RemoveContainer" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" Oct 30 21:13:37.540571 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:37.540515 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:37.542391 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.542370 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qw92" event={"ID":"06a00eb6-ceaa-49ee-8a17-01ba1009767e","Type":"ContainerStarted","Data":"612c2b7224e6bd6f53f13ec394e0430a167366b89ed03a617874c673e097cd45"} Oct 30 21:13:37.542515 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.542496 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:37.614777 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.614741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:37.657942 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.657899 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" podStartSLOduration=344.285372435 podStartE2EDuration="5m51.657878877s" totalImagesPullingTime="7.372506442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:37.657270516 +0000 UTC m=+40.876078346" watchObservedRunningTime="2026-10-30 21:13:37.657878877 +0000 UTC m=+40.876686707" Oct 30 21:13:37.693763 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.693728 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.885292 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.885202 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.894596 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:37.894548 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2qw92" podStartSLOduration=35.167659999 podStartE2EDuration="39.894535282s" totalImagesPullingTime="4.726875283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:37.8936532 +0000 UTC m=+41.112461032" watchObservedRunningTime="2026-10-30 21:13:37.894535282 +0000 UTC m=+41.113343107" Oct 30 21:13:38.026196 ip-10-0-132-23 kubenswrapper[2583]: W1030 21:13:38.026151 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bd86df_4448_46c9_aac0_2e8496b22533.slice/crio-87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7 WatchSource:0}: Error finding container 87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7: Status 404 returned error can't find the container with id 87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7 Oct 30 21:13:38.028071 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.028048 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd"] Oct 30 21:13:38.069617 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.069532 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:38.546600 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.546559 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" event={"ID":"d0bd86df-4448-46c9-aac0-2e8496b22533","Type":"ContainerStarted","Data":"87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7"} Oct 30 21:13:38.638459 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.638433 2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:38.638572 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.638475 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:38.638772 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:38.638760 2583 scope.go:122] "RemoveContainer" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" Oct 30 21:13:38.638911 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:38.638895 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:39.080316 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:39.080289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:39.080407 ip-10-0-132-23 kubenswrapper[2583]: I1030 21:13:39.080349 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:39.080469 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:39.080425 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:39.080469 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:39.080435 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:39.080560 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:39.080470 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:43.080458917 +0000 UTC m=+46.299266726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:39.080560 ip-10-0-132-23 kubenswrapper[2583]: E1030 21:13:39.080495 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:43.080478549 +0000 UTC m=+46.299286371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:39.253271 ip-10-0-132-23 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:13:39.265082 ip-10-0-132-23 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:13:39.265256 ip-10-0-132-23 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:13:39.265304 ip-10-0-132-23 systemd[1]: kubelet.service: Consumed 2.920s CPU time. Oct 30 21:13:39.276393 ip-10-0-132-23 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:13:39.334619 ip-10-0-132-23 kubenswrapper[4918]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:39.334619 ip-10-0-132-23 kubenswrapper[4918]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:13:39.334619 ip-10-0-132-23 kubenswrapper[4918]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:39.334619 ip-10-0-132-23 kubenswrapper[4918]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:39.336731 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336645 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.336731 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336730 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.336731 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336734 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336737 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336740 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336743 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336745 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336748 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336751 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336753 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336756 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336759 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336762 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336765 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336767 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336777 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336800 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336804 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336807 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336811 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336814 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336816 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.336825 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336819 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336822 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336825 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336828 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336831 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336833 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336837 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336840 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336843 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336845 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336848 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336850 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336853 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336856 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336858 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336861 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336863 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336866 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336869 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.337300 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336871 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336875 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336880 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336882 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336885 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336889 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336891 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336894 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336897 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336900 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336902 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336905 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336907 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336911 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336914 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336917 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336920 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336922 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336925 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.337776 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336928 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336931 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336933 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336936 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336939 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336941 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336944 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336946 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336949 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336951 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336954 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336957 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336960 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336962 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336965 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336967 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336970 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336972 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336975 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336978 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.338275 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336980 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336983 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336986 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336988 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336991 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336994 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.336997 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337000 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337002 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337005 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337008 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337011 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337016 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337019 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337022 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337025 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337028 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337030 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337033 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.338775 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337035 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337038 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337041 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337044 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337046 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337050 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337053 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337859 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337866 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337869 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337872 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337875 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337877 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337894 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337899 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337903 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337907 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337920 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337925 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337931 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.339266 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337933 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337936 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337938 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337941 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337944 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337947 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337949 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337952 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337954 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337957 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337960 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337963 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337965 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337968 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337970 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337973 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337977 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337982 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337985 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.339774 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337988 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337991 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337994 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337996 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.337999 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338002 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338004 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338014 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338017 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338020 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338023 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338026 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338030 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338033 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338036 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338038 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338041 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338044 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338046 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.340292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338049 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338057 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338060 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338062 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338065 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338068 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338070 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338073 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338075 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338078 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338081 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338083 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338086 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338089 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338091 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338094 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338097 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338099 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338102 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338105 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.340784 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338107 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338117 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338120 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338122 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338125 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338129 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338131 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338134 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338137 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338139 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338142 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338146 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338149 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338152 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338155 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338158 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338161 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338163 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338166 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.341313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338168 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338171 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338174 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338176 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338179 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338181 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338184 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338186 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338189 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338191 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338194 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338197 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338199 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338203 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338205 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338214 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338771 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338777 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338782 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.341803 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338789 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338792 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338794 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338797 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338800 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338803 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338806 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338809 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338811 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338814 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338816 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338819 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338828 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338831 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338833 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338836 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338840 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338843 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338847 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.342293 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338851 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338853 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338856 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338859 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338861 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338864 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338866 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338869 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338871 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338880 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338883 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338885 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338890 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338892 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338895 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338897 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338900 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338903 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338905 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338908 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.342806 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338911 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338913 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338916 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338918 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338921 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338924 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338926 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338929 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338931 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338934 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338937 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338939 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338942 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338944 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338946 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338949 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338952 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338954 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338957 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.343295 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338960 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338962 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338965 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338973 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338976 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338980 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338983 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338987 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338990 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338993 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338996 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.338998 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339001 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339004 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339007 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339009 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339012 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339015 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339017 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.343765 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339020 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339030 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339033 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339036 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339039 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339042 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339044 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339047 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339050 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339052 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339055 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339057 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339060 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339063 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339065 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339068 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339070 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339078 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339083 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339086 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.344259 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339089 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339091 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339094 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339097 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339099 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339102 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339197 4918 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339204 4918 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339238 4918 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339244 4918 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339248 4918 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339252 4918 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339256 4918 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339264 4918 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339269 4918 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339272 4918 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339276 4918 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339279 4918 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339283 4918 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339286 4918 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339289 4918 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339292 4918 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339295 4918 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339298 4918 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:13:39.344758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339304 4918 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339307 4918 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339311 4918 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339314 4918 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339317 4918 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339321 4918 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339330 4918 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339334 4918 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339337 4918 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339340 4918 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339343 4918 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339346 4918 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339349 4918 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339352 4918 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339357 4918 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339360 4918 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339363 4918 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339366 4918 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339369 4918 flags.go:64] FLAG: --enable-server="true" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339371 4918 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339379 4918 flags.go:64] FLAG: --event-burst="100" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339382 4918 flags.go:64] FLAG: --event-qps="50" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339386 4918 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339390 4918 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:13:39.345628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339393 4918 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339397 4918 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339400 4918 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339403 4918 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339406 4918 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339409 4918 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339412 4918 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339415 4918 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339418 4918 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339421 4918 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339424 4918 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339427 4918 flags.go:64] FLAG: --feature-gates="" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339431 4918 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339434 4918 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339437 4918 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339446 4918 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339456 4918 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339459 4918 flags.go:64] FLAG: --help="false" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339463 4918 flags.go:64] FLAG: --hostname-override="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339467 4918 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339470 4918 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339473 4918 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339476 4918 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:13:39.346435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339480 4918 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339483 4918 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339486 4918 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339489 4918 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339491 4918 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339494 4918 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339498 4918 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339502 4918 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339505 4918 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339508 4918 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339511 4918 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339514 4918 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339517 4918 flags.go:64] FLAG: --lock-file="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339520 4918 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339523 4918 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339526 4918 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339532 4918 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339535 4918 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339538 4918 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339541 4918 flags.go:64] FLAG: --logging-format="text" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339544 4918 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339548 4918 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339550 4918 flags.go:64] FLAG: --manifest-url="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339553 4918 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339558 4918 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:13:39.347032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339568 4918 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339572 4918 flags.go:64] FLAG: --max-pods="110" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339575 4918 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339578 4918 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339581 4918 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339584 4918 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339587 4918 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339590 4918 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339593 4918 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339599 4918 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339603 4918 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339606 4918 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339609 4918 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339612 4918 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339616 4918 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339619 4918 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339622 4918 flags.go:64] FLAG: --port="10250" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339626 4918 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339629 4918 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-009105aa62ae44ccc" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339632 4918 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339635 4918 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339638 4918 flags.go:64] FLAG: --register-node="true" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339641 4918 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339645 4918 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339648 4918 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339650 4918 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:13:39.347672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339653 4918 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339657 4918 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339660 4918 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339663 4918 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339666 4918 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339669 4918 flags.go:64] FLAG: --runonce="false" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339672 4918 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339681 4918 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339684 4918 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339687 4918 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339690 4918 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339693 4918 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339696 4918 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339699 4918 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339716 4918 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339721 4918 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339726 4918 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339730 4918 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339734 4918 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339737 4918 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339742 4918 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339748 4918 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339752 4918 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339755 4918 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339762 4918 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:13:39.348321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339765 4918 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339768 4918 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339770 4918 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339774 4918 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339777 4918 flags.go:64] FLAG: --v="2" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339781 4918 flags.go:64] FLAG: --version="false" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339786 4918 flags.go:64] FLAG: --vmodule="" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339790 4918 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.339793 4918 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339992 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.339998 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340002 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340004 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340007 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340010 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340018 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340022 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340024 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340027 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340029 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340032 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.349041 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340035 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340038 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340040 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340043 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340045 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340048 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340052 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340055 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340059 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340062 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340064 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340067 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340070 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340072 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340075 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340077 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340080 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340083 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340085 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.349545 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340088 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340092 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340095 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340098 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340101 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340104 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340107 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340110 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340119 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340122 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340125 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340127 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340130 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340133 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340135 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340138 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340140 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340143 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340145 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.350094 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340149 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340152 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340156 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340158 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340161 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340164 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340166 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340169 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340171 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340174 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340176 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340179 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340182 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340184 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340187 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340189 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340192 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340195 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340199 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.350624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340203 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340207 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340210 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340220 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340223 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340226 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340229 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340232 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340235 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340237 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340240 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340242 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340244 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340248 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340251 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340255 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340258 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340261 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340263 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340266 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.351207 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340269 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340271 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340274 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340276 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340279 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340282 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340285 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340287 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340290 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340293 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340296 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340298 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340301 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340304 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340307 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340310 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.351780 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340318 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.340344 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340385 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340390 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340393 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340396 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340399 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340401 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340404 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340408 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340411 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340414 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340419 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340421 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340424 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340427 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.352190 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340429 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340432 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340440 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340444 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340446 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340449 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340452 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340454 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340457 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340460 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340462 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340465 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340467 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340470 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340473 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340475 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340478 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340486 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340489 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.352606 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340491 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340494 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340497 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340499 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340502 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340504 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340509 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340512 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340514 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340518 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340521 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340524 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340526 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340529 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340532 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340534 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340539 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340543 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340546 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.353182 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340548 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340551 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340554 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340557 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340559 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340562 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340564 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340567 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340570 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340572 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340575 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340577 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340585 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340588 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340591 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340593 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340596 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340598 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340601 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.353721 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340605 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340608 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340610 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340615 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340617 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340620 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340622 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340625 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340627 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340630 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340633 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340635 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340638 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340640 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340644 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340648 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340651 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340654 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340657 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.354246 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340660 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340663 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340667 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340669 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340672 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340674 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340677 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340686 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340689 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340692 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340695 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340698 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340700 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340720 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340723 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.340725 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.344852 4918 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.344972 4918 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.344998 4918 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.345006 4918 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:13:39.354800 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345333 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345351 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345357 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345363 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345368 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345373 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345378 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345382 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345386 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345390 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345395 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345400 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345404 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345408 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345418 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345423 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345427 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345433 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345440 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.355281 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345446 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345451 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345455 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345460 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345464 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345468 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345473 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345478 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345487 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345491 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345529 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345534 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345539 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345544 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345548 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345553 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345558 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345562 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345566 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345571 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.355760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345581 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345586 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345591 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345595 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345601 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345606 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345610 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345614 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345619 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345623 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345627 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345657 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345662 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345671 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345676 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345680 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345684 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345689 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345693 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345697 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.356429 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345701 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345728 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345733 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345762 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345767 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345771 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345780 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345784 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345789 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345793 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345798 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345802 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345807 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345811 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345816 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345820 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345825 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345830 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345839 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.357307 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345843 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345847 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345851 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345856 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345860 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345865 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345869 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345874 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345878 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345882 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345892 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345896 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345907 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345913 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345918 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.345922 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346008 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346156 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346164 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.358154 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346168 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346173 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346176 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346179 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346182 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346185 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346187 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346190 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346195 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.346233 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346393 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346397 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346400 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346403 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346406 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346409 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.359025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346412 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346415 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346417 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346420 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346423 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346425 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346428 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346431 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346434 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346437 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346440 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346443 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346446 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346449 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346452 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346455 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346458 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346461 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346463 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346466 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.359688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346469 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346471 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346474 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346476 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346479 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346482 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346484 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346486 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346489 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346492 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346494 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346497 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346499 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346502 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346504 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346507 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346510 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346512 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346515 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346518 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.360575 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346521 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346524 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346527 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346531 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346535 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346538 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346541 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346545 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346547 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346569 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346574 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346577 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346581 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346584 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346587 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346590 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346592 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346595 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346598 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.361483 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346600 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346603 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346606 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346608 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346611 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346613 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346616 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346619 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346622 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346625 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346628 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346630 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346633 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346636 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346640 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346644 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346647 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346650 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346652 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.362391 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346655 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346658 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346660 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346663 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346666 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346668 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346671 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346673 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346676 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346678 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346681 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346683 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346686 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346688 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346691 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346693 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346696 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346699 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346701 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.363356 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346724 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346728 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346731 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.346756 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346788 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346793 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346797 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346800 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346803 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346806 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346809 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346812 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346815 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346817 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346820 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346823 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:39.364232 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346826 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346828 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346831 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346834 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346837 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346840 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346842 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346845 4918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346847 4918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346850 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346853 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346855 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346858 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346860 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346863 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346866 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346868 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346871 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346874 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346877 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:39.364995 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346879 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346882 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346884 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346887 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346890 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346892 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346895 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346898 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346900 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346904 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346907 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346910 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346912 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346916 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346920 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346923 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346926 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346929 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346932 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:39.365987 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346934 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346937 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346939 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346943 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346945 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346948 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346951 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346953 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346956 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346959 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346961 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346964 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346967 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346970 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346973 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346975 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346978 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346981 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346983 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346986 4918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:39.366842 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346988 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346991 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346993 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.346996 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347000 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347003 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347007 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347010 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347012 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347018 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347021 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347024 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347028 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347030 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347033 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347035 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347038 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347041 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347044 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:39.367661 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347046 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347049 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347052 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347054 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347057 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347060 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347063 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347065 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347068 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347070 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347073 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347076 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347079 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347081 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347084 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:39.347086 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.347142 4918 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348134 4918 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:13:39.368202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348187 4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348363 4918 server.go:1014] "Starting client certificate rotation" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348514 4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348574 4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:07:58 +0000 UTC" deadline="2028-04-22 04:28:31.628128987 +0000 UTC" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348615 4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12943h14m52.279517415s" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.348786 4918 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.349757 4918 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.350999 4918 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.352665 4918 log.go:25] "Validated CRI v1 image API" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.353330 4918 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:13:39.368641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.355071 4918 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d60a68f4-b920-4a42-90cc-dab0b95bc8e5:/dev/nvme0n1p3 f17da894-7b78-43b9-9bd6-f6945ecf3e44:/dev/nvme0n1p4] Oct 30 21:13:39.369144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.355091 4918 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/431dc86ba2f6dd72c2dc27b50ba3b404cb6f4a0fc1333cded0785fb4c4585ea8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/431dc86ba2f6dd72c2dc27b50ba3b404cb6f4a0fc1333cded0785fb4c4585ea8/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f6fc9fcf1c9d763d0ebb6bc751d120f84f3780df2753ffa426f8707eed19c7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f6fc9fcf1c9d763d0ebb6bc751d120f84f3780df2753ffa426f8707eed19c7a/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0217fb37-bb64-4dd6-aa3f-480891bd47dc/volumes/kubernetes.io~projected/kube-api-access-chwdb:{mountpoint:/var/lib/kubelet/pods/0217fb37-bb64-4dd6-aa3f-480891bd47dc/volumes/kubernetes.io~projected/kube-api-access-chwdb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a00eb6-ceaa-49ee-8a17-01ba1009767e/volumes/kubernetes.io~projected/kube-api-access-5mzxp:{mountpoint:/var/lib/kubelet/pods/06a00eb6-ceaa-49ee-8a17-01ba1009767e/volumes/kubernetes.io~projected/kube-api-access-5mzxp major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08289af5-6136-4988-9455-9f2eef24caef/volumes/kubernetes.io~projected/kube-api-access-8c4k6:{mountpoint:/var/lib/kubelet/pods/08289af5-6136-4988-9455-9f2eef24caef/volumes/kubernetes.io~projected/kube-api-access-8c4k6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60/volumes/kubernetes.io~projected/kube-api-access-8kn4c:{mountpoint:/var/lib/kubelet/pods/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60/volumes/kubernetes.io~projected/kube-api-access-8kn4c major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/kube-api-access-8jph5:{mountpoint:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/kube-api-access-8jph5 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~projected/kube-api-access-h2blx:{mountpoint:/var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~projected/kube-api-access-h2blx major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2da5bf9f-3892-4b54-a337-880b1e19faf8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2da5bf9f-3892-4b54-a337-880b1e19faf8/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/kube-api-access-6r2wd:{mountpoint:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/kube-api-access-6r2wd major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/323fa83f-8f63-4a65-9837-d31e6f6de11d/volumes/kubernetes.io~projected/kube-api-access-tsz6j:{mountpoint:/var/lib/kubelet/pods/323fa83f-8f63-4a65-9837-d31e6f6de11d/volumes/kubernetes.io~projected/kube-api-access-tsz6j major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~projected/kube-api-access-x9mcn:{mountpoint:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~projected/kube-api-access-x9mcn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585cbecc-cf75-4689-979e-3b70dd6cd0b3/volumes/kubernetes.io~projected/kube-api-access-flq22:{mountpoint:/var/lib/kubelet/pods/585cbecc-cf75-4689-979e-3b70dd6cd0b3/volumes/kubernetes.io~projected/kube-api-access-flq22 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71cd5ea6-5cf5-46b7-a778-7a2921fbd837/volumes/kubernetes.io~projected/kube-api-access-fz6r5:{mountpoint:/var/lib/kubelet/pods/71cd5ea6-5cf5-46b7-a778-7a2921fbd837/volumes/kubernetes.io~projected/kube-api-access-fz6r5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88f63d46-40a6-46c0-b359-d58509ccba89/volumes/kubernetes.io~projected/kube-api-access-l2k4d:{mountpoint:/var/lib/kubelet/pods/88f63d46-40a6-46c0-b359-d58509ccba89/volumes/kubernetes.io~projected/kube-api-access-l2k4d major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~projected/kube-api-access-hsmgt:{mountpoint:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~projected/kube-api-access-hsmgt major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c729fe8-4a69-477a-a247-1e750d51c96a/volumes/kubernetes.io~projected/kube-api-access-nnsdm:{mountpoint:/var/lib/kubelet/pods/9c729fe8-4a69-477a-a247-1e750d51c96a/volumes/kubernetes.io~projected/kube-api-access-nnsdm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad304cde-8b80-44b3-8631-515bbd7b80fc/volumes/kubernetes.io~projected/kube-api-access-4n259:{mountpoint:/var/lib/kubelet/pods/ad304cde-8b80-44b3-8631-515bbd7b80fc/volumes/kubernetes.io~projected/kube-api-access-4n259 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0bd86df-4448-46c9-aac0-2e8496b22533/volumes/kubernetes.io~projected/kube-api-access-bz72g:{mountpoint:/var/lib/kubelet/pods/d0bd86df-4448-46c9-aac0-2e8496b22533/volumes/kubernetes.io~projected/kube-api-access-bz72g major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d/volumes/kubernetes.io~projected/kube-api-access-kxb8f:{mountpoint:/var/lib/kubelet/pods/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d/volumes/kubernetes.io~projected/kube-api-access-kxb8f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7cd2ff9-7aeb-4ebf-9605-a213958c1153/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d7cd2ff9-7aeb-4ebf-9605-a213958c1153/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dce12b75-7554-47a5-8214-f40f414060d7/volumes/kubernetes.io~projected/kube-api-access-9xr8r:{mountpoint:/var/lib/kubelet/pods/dce12b75-7554-47a5-8214-f40f414060d7/volumes/kubernetes.io~projected/kube-api-access-9xr8r major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~projected/kube-api-access-jxs2f:{mountpoint:/var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~projected/kube-api-access-jxs2f major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~projected/kube-api-access-2pcfk:{mountpoint:/var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~projected/kube-api-access-2pcfk major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfd123e3-d34f-411d-a77f-d147f5e3368e/volumes/kubernetes.io~projected/kube-api-access-trtgb:{mountpoint:/var/lib/kubelet/pods/dfd123e3-d34f-411d-a77f-d147f5e3368e/volumes/kubernetes.io~projected/kube-api-access-trtgb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e498e377-392b-421d-a455-1f7624968e6a/volumes/kubernetes.io~projected/kube-api-access-6ksvg:{mountpoint:/var/lib/kubelet/pods/e498e377-392b-421d-a455-1f7624968e6a/volumes/kubernetes.io~projected/kube-api-access-6ksvg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa/volumes/kubernetes.io~projected/kube-api-access-gdhph:{mountpoint:/var/lib/kubelet/pods/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa/volumes/kubernetes.io~projected/kube-api-access-gdhph major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~projected/kube-api-access-662r6:{mountpoint:/var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~projected/kube-api-access-662r6 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~projected/kube-api-access-tpv4q:{mountpoint:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~projected/kube-api-access-tpv4q major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~projected/kube-api-access-tqbwr:{mountpoint:/var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~projected/kube-api-access-tqbwr major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2becdfd-f3fd-4a40-8278-566d27fc1ebb/volumes/kubernetes.io~projected/kube-api-access-bwvrw:{mountpoint:/var/lib/kubelet/pods/f2becdfd-f3fd-4a40-8278-566d27fc1ebb/volumes/kubernetes.io~projected/kube-api-access-bwvrw major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9771a55-0c02-4b30-9fe7-1850bd414cde/volumes/kubernetes.io~projected/kube-api-access-7w28n:{mountpoint:/var/lib/kubelet/pods/f9771a55-0c02-4b30-9fe7-1850bd414cde/volumes/kubernetes.io~projected/kube-api-access-7w28n major:0 minor:219 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a7d8c63817bb1dcada562f6a858951fb996a18c4b20608923fa5e49d2308f369/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b121b0341becaa156c1c752604d01715c618731f025860259d6ffccba4eef90a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b4f79a974d70a30dade6c162740698a6d39e0ac2f9b4503db3007cd8346ddf14/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/82d5de0adb926eb5dd696ffac036a6a64d12417c9641b9f53d2214e7b738c51c/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/ae9f918131156e4ab55ece122e24ac8661d524c6e625f5589b08875f04c6af14/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/221af8cc03484a2cb9653a038a9760d98789337836ceb49b834f022b1e7f6f65/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b0e2347442b704a9df74108bc5735063fabd65d3c07222bfa2d94637a88ed34f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e30811082ba4cc30ca8ce6202b162664f40c1868c31535656ee26a6e2a582c3a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e703058c0edf37c9fc7b2f4e130fd6bd2aefb57ef4f75c06779677a5c86d6bcc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/43cca79da7f664ffdc6f95f3f042aa4288fd66e6e275ded77eefdd7ccbad9cb7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5a613d61aa7a01cdd08e38042d8dd58d5ed03cd8161a13b4f531f1caaf5c43a2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2c29c3ec66966d3c20f4a557d56bb2d2bf9d661b11bc336770e8ac871987e2c0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0db2041c13023387f245f71d14a8f964558c5cdd208986c5524815d885b554b2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5afeade77eac850e2858cc532bf6106af32bb0135dd87c47066cbd89363cba61/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c41ea1f80a67e974d0c76d9026f13cc8c3461385d5f34775806af97ea6c4b250/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e234f4c5ab95a388496fab945eede4054579e96c1390eaf4ca2ef02c6044726a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dddacee504db8f034aef306beb667cc4f0b50f12dc557c8771dd2c7705bdfb18/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/eaefeb4162869063b7b36e1ff894b93b47ef26b7a888a96c5470bd87d26754fd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/355a04b5d7c2c741f9d31f71676afe61bf589412e69ee1816adc3624985e7c61/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1802d9be5bb65cc210cdc286ce9a8cefc57ac7e7e37e29e9736c6f1daa7f3718/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/39b28d696ec2e5586b606669c985f273f656b8e26a71452d362e4f392f300aed/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f6857ebd815c4c99cb90c88c0e4736f5914681786d423676dae14e3f5de92909/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/e7b7817579bf7f1b038f96cf53762de722258e7b4217afc42394cb02d56052d5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/91e05ee066802c4cfd33c1d4d45f5f9e0f046ffe492d2c7efa017f233cc119ff/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/52047bd29e9f88031f34028dca0d7b12ef5815a750d7460bec1c068b6df76d5e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/581f28abb33d0cb1a5e6826267d988c1ef628f89840ef51314c35c8f25ab9d54/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/bed638fe26acaae5e6ea05d9865e22b286688c104b4e6c3ffee2f3b5ffa21f67/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d68abda218c4336d85e0a5b97c02aaa5f7e1b5509873f83c8444cf3e96e058cd/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e33a82338e4464714eb1af166882c3541c0ea9eeafd300672ca3980ab39e6464/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/068332382d47a2aaa38f271ed86af1987ebbdfa84c74b88e32b78e8651bb8daa/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/cdcaa5f69785c592999b47cf67f8ac6c6ac49d700abe7629980097ae64900db1/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/95a8870122e879ac94c8bb14c7c1f09e23153c98f0cf4d3d32c145cc56a9559c/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/3d442285061dab5598004eb5e5d63d709be7aec72d59d8e135a9b2cad1297cad/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/88c1566793d7191455befae864235eae0e6c3e0ab4d5196178c42304321705a3/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/698cd4aef72e0c0074859d22cc0ab8160e95e4ad4cd59ef6177c123c215d62b2/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/ad5f8715713b3fa88c51d1bbed5f22228aa0a4e3b1b3d9831db47a7aec61fccb/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8fa16272b4df7ff4c72a4e5c8532f78b8c4a66556de6ae8743cf5bfc75b66be9/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0c07154d18023abf886acd7a1449600bd58d42136c1bcb0145d4ad6b08542c78/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/67f06b9a07f694c5c87a17f303a1d2dac233f8ff9e8d77073b2210e3e228470a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/fbf1728a7c0b659056f33a5157a56ccefcfe27a340d343034105a6638bdc77e5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/0d5d0668633599a0b93606bc18d2af3bbd631fe36b65c8f094194180dc6914c1/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/16ce0f47d3a04f02952a20dbf9268a043090bf45043b882e2bb366b37a89c263/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/417b8c2497d2baa74bfaa6cf297858a7a671866264226373c12317d0891e5f03/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/62054885589b35087ef20324ca353862c3cf67e0e72227858379faa17fd4653b/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/1187de4df5aa6572d41114e57cdbc76e09907d66bba542896c98bac6d5684349/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c6b1e25ef704130f5738307175c6a3773b252a14041a9a529f8f2e1ec4a18cd4/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/46b33d0e98b1748f2285b930c67ba008d97f6853157d131415253ed0ebcd7af5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dd05bcaf6d7ca99b20dfe3b1797a653f98f5f99e4845b9302afd40e9b510a2ea/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6d290c4beca4cf54ac3f5184d934553723b037e3558a41d8470f94c4f65ae5f6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/3fc1d27caa4a9c1979042a309d448569ef70d9344cea9466ac6f8ae7396db7b9/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b7015a14bb79e1e584ebe2e59337f9523de126faa026d96846f420abd3c07cda/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d69b97ff10acd1e2235105c4a24658df364b7f106373ced8e8a10cfc62185ebd/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ce87750a7eae685b981b3b1b6151363c195cb0f977fa18d17768f025506756cb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ae5f68ceb81c66245f1fd8dee054af4991ede596823d02b807f153ca309a9a07/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/dd92df15afd5e4790f77150a73f9ba23d3d5673fedfa8497c940b722ef893032/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.367505 4918 manager.go:217] Machine: {Timestamp:2026-10-30 21:13:39.366517028 +0000 UTC m=+0.059999259 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099749 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258f8a64012d40e3125b5487fb8835 SystemUUID:ec258f8a-6401-2d40-e312-5b5487fb8835 BootID:31d47b79-e27d-453f-a0ed-b1ef58f03b1c Filesystems:[{Device:/run/containers/storage/overlay-containers/ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa/volumes/kubernetes.io~projected/kube-api-access-gdhph DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f2becdfd-f3fd-4a40-8278-566d27fc1ebb/volumes/kubernetes.io~projected/kube-api-access-bwvrw DeviceMajor:0 DeviceMinor:239 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d0bd86df-4448-46c9-aac0-2e8496b22533/volumes/kubernetes.io~projected/kube-api-access-bz72g DeviceMajor:0 DeviceMinor:327 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7cd2ff9-7aeb-4ebf-9605-a213958c1153/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88f63d46-40a6-46c0-b359-d58509ccba89/volumes/kubernetes.io~projected/kube-api-access-l2k4d DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d/volumes/kubernetes.io~projected/kube-api-access-kxb8f DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2828a991aafe64873080d7011f290be2e9475033adaa4be4a007cb4e5877280d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e498e377-392b-421d-a455-1f7624968e6a/volumes/kubernetes.io~projected/kube-api-access-6ksvg DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e6f781a4f36e10c2fe1651deadd24d21ba5d7b5b7564ec94ab5cea7f7cbb9d5/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5246bef503b5213894ac55e9aaee46ade62c0a3212fbc08cca99f25f35bf5caa/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~projected/kube-api-access-h2blx DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dce12b75-7554-47a5-8214-f40f414060d7/volumes/kubernetes.io~projected/kube-api-access-9xr8r DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~projected/kube-api-access-hsmgt DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/71cd5ea6-5cf5-46b7-a778-7a2921fbd837/volumes/kubernetes.io~projected/kube-api-access-fz6r5 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/kube-api-access-6r2wd DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8c379f5e2e53b36f508f1e1595718e350932289b6241ea79e5a4a50f8a05a95/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c729fe8-4a69-477a-a247-1e750d51c96a/volumes/kubernetes.io~projected/kube-api-access-nnsdm DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/kube-api-access-8jph5 DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/274417e7-ffd3-4de2-a249-a15e27177905/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~projected/kube-api-access-tqbwr DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e95e0daf-f36a-4951-8ead-9cf4ef55a928/volumes/kubernetes.io~projected/kube-api-access-662r6 DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32b23380b9f58bf62a9602c0abfd04259606b947d07aea3b60acb8534dbb05db/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~projected/kube-api-access-jxs2f DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59ca1ce61234ce3433ea7d1a3eaf614bc85a9e769832ba30f246e4cfdf4e2c67/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9771a55-0c02-4b30-9fe7-1850bd414cde/volumes/kubernetes.io~projected/kube-api-access-7w28n DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/08289af5-6136-4988-9455-9f2eef24caef/volumes/kubernetes.io~projected/kube-api-access-8c4k6 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec41e273-2c37-467e-b55d-cef56c254d57/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6f21caa2b4542d7211c142e5ce7a092cd2bb2d12d14275794c2cde63377ad97/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f6fc9fcf1c9d763d0ebb6bc751d120f84f3780df2753ffa426f8707eed19c7a/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de5cba4a-8119-475d-a365-f7977405ea6e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0217fb37-bb64-4dd6-aa3f-480891bd47dc/volumes/kubernetes.io~projected/kube-api-access-chwdb DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7a69c4-438f-4679-9255-d0cae7ffc19f/volumes/kubernetes.io~projected/kube-api-access-x9mcn DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fb63c5230281fc7b7459e5562745e013789c3f7a8f440098611bc35f1837c36/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87e5138271c824bb1c320c4b78b205bb78875b6ed1b9e40b95d7c6a52fb446a7/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/585cbecc-cf75-4689-979e-3b70dd6cd0b3/volumes/kubernetes.io~projected/kube-api-access-flq22 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ef71d2ed41f6d6a94eda02c2310f5927bbf93bc3b25598189781597f09ca4a6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed3d99968fa6a4e6a0e8a3744629c901588b90adc4264077729d21b10a02ca28/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a66bdef73808019b966ed2fc73c9401708f3fb6baf55678026e009d7e31e930/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6f954a071363b16a7cf5587c2d45ea83a19df7d6097dba203c81a96823bee79/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df904ba1-72fc-41f3-82d7-959374abe111/volumes/kubernetes.io~projected/kube-api-access-2pcfk DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94567a77-bf25-464b-adde-cfb50ccfea99/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fd1207e6ad72307f558807e0921639e46366a513176b6f40892385d973845a9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9db3775e3778239c410b5c0d3796aa83e3117761e9f5997d5ac983f0e5f2706/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/431dc86ba2f6dd72c2dc27b50ba3b404cb6f4a0fc1333cded0785fb4c4585ea8/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d00fd6f77c883b00d5addae4995c66afd5c430e14ad31eacb07ebf2b5355d1f/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1b9b381ce3fb2f57803e0b9275bf5c78fdef94dc5abc084f049e008aaf93ac1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f6483a819928fe5d88c2e9c2b34f54658b30c838ec300488aa13beb088f704d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f86f24cd5c45a6672af2234abd3d5bbc74256fc3344e9016fc737cce124751b5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/06a00eb6-ceaa-49ee-8a17-01ba1009767e/volumes/kubernetes.io~projected/kube-api-access-5mzxp DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dfd123e3-d34f-411d-a77f-d147f5e3368e/volumes/kubernetes.io~projected/kube-api-access-trtgb DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60/volumes/kubernetes.io~projected/kube-api-access-8kn4c DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad304cde-8b80-44b3-8631-515bbd7b80fc/volumes/kubernetes.io~projected/kube-api-access-4n259 DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2da5bf9f-3892-4b54-a337-880b1e19faf8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1/volumes/kubernetes.io~projected/kube-api-access-tpv4q DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/323fa83f-8f63-4a65-9837-d31e6f6de11d/volumes/kubernetes.io~projected/kube-api-access-tsz6j DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1e6f781a4f36e10 MacAddress:be:70:e7:9c:82:fc Speed:10000 Mtu:8901} {Name:32b23380b9f58bf MacAddress:1a:a8:c6:3d:fd:a4 Speed:10000 Mtu:8901} {Name:431dc86ba2f6dd7 MacAddress:82:9f:2f:ec:70:b8 Speed:10000 Mtu:8901} {Name:5246bef503b5213 MacAddress:9a:ac:62:33:9f:ba Speed:10000 Mtu:8901} {Name:59ca1ce61234ce3 MacAddress:b6:2f:af:9a:68:62 Speed:10000 Mtu:8901} {Name:5ef71d2ed41f6d6 MacAddress:4a:77:16:9d:45:c2 Speed:10000 Mtu:8901} {Name:87e5138271c824b MacAddress:0e:2d:91:8c:93:93 Speed:10000 Mtu:8901} {Name:8f6fc9fcf1c9d76 MacAddress:5e:bd:8e:73:f6:15 Speed:10000 Mtu:8901} {Name:9d00fd6f77c883b MacAddress:6e:f8:00:c5:65:3b Speed:10000 Mtu:8901} {Name:9fd1207e6ad7230 MacAddress:56:32:ca:98:fe:37 Speed:10000 Mtu:8901} {Name:b8c379f5e2e53b3 MacAddress:12:f0:59:dc:e0:84 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:db:02:f2:29 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:30:71:33:ce:db Speed:0 Mtu:8901} {Name:c6f21caa2b4542d MacAddress:4e:e9:6e:69:67:fa Speed:10000 Mtu:8901} {Name:ed3d99968fa6a4e MacAddress:d2:4d:ea:f4:43:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:db:02:f2:29 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2e:14:33:b7:4b:b1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:94:78:2b:f8:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.368544 4918 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.368602 4918 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.368891 4918 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.368917 4918 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-23.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369035 4918 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369044 4918 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369058 4918 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369071 4918 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369089 4918 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369243 4918 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:13:39.369367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369350 4918 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:13:39.370023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369387 4918 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:13:39.370023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369402 4918 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:13:39.370023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369414 4918 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:13:39.370023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369423 4918 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:13:39.370023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.369430 4918 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:13:39.370332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.370316 4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:13:39.370374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.370333 4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:39.370793 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.370778 4918 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:13:39.381852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.381834 4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:13:39.381852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.381847 4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:39.381972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.381964 4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:13:39.382171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382157 4918 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:13:39.382200 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382194 4918 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:13:39.382308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382298 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:13:39.382340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382313 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:13:39.382340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382322 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:13:39.382340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382332 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:13:39.382340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382341 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382347 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382353 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382359 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382366 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382372 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382386 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382396 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382429 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:13:39.382469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.382436 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:13:39.385757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.385744 4918 server.go:1263] "Started kubelet" Oct 30 21:13:39.387247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.385845 4918 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:13:39.387247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.386142 4918 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:13:39.387247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.386225 4918 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:13:39.387247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.386484 4918 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:13:39.387247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.387250 4918 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:13:39.386257 ip-10-0-132-23 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:13:39.391215 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391196 4918 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:13:39.391352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391218 4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:13:39.391352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391270 4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:07:59 +0000 UTC" deadline="2028-06-30 11:52:57.612024628 +0000 UTC" Oct 30 21:13:39.391352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391289 4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14606h39m18.220738619s" Oct 30 21:13:39.391352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391321 4918 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:13:39.391352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391329 4918 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:13:39.391691 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.391352 4918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:13:39.391691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.391421 4918 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:13:39.393945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.393914 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0217fb37-bb64-4dd6-aa3f-480891bd47dc" volumeName="kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config" seLinuxMountContext="" Oct 30 21:13:39.394014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.393947 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a00eb6-ceaa-49ee-8a17-01ba1009767e" volumeName="kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp" seLinuxMountContext="" Oct 30 21:13:39.394014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.393962 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71cd5ea6-5cf5-46b7-a778-7a2921fbd837" volumeName="kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5" seLinuxMountContext="" Oct 30 21:13:39.394014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.393976 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c729fe8-4a69-477a-a247-1e750d51c96a" volumeName="kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:13:39.394014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.393990 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0bd86df-4448-46c9-aac0-2e8496b22533" volumeName="kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g" seLinuxMountContext="" Oct 30 21:13:39.394014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394003 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d" volumeName="kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394016 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dce12b75-7554-47a5-8214-f40f414060d7" volumeName="kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394029 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08289af5-6136-4988-9455-9f2eef24caef" volumeName="kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394044 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8bf47b-05dc-4987-a7e9-7fa682c8ab60" volumeName="kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394057 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5cba4a-8119-475d-a365-f7977405ea6e" volumeName="kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394070 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394083 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394097 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394128 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94567a77-bf25-464b-adde-cfb50ccfea99" volumeName="kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394143 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394153 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394161 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9771a55-0c02-4b30-9fe7-1850bd414cde" volumeName="kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394168 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad304cde-8b80-44b3-8631-515bbd7b80fc" volumeName="kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394184 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:13:39.394198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394193 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274417e7-ffd3-4de2-a249-a15e27177905" volumeName="kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394204 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394217 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394231 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7a69c4-438f-4679-9255-d0cae7ffc19f" volumeName="kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394244 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585cbecc-cf75-4689-979e-3b70dd6cd0b3" volumeName="kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394256 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df904ba1-72fc-41f3-82d7-959374abe111" volumeName="kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394269 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71cd5ea6-5cf5-46b7-a778-7a2921fbd837" volumeName="kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394278 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08289af5-6136-4988-9455-9f2eef24caef" volumeName="kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394286 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8bf47b-05dc-4987-a7e9-7fa682c8ab60" volumeName="kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394296 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585cbecc-cf75-4689-979e-3b70dd6cd0b3" volumeName="kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394311 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394319 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a2ed00-d2f6-45cd-9085-7e93ce8031e1" volumeName="kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394326 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394332 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394339 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394348 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5cba4a-8119-475d-a365-f7977405ea6e" volumeName="kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394355 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfd123e3-d34f-411d-a77f-d147f5e3368e" volumeName="kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394363 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7a69c4-438f-4679-9255-d0cae7ffc19f" volumeName="kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394370 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394377 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a2ed00-d2f6-45cd-9085-7e93ce8031e1" volumeName="kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394383 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394391 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad304cde-8b80-44b3-8631-515bbd7b80fc" volumeName="kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394399 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7cd2ff9-7aeb-4ebf-9605-a213958c1153" volumeName="kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394406 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df904ba1-72fc-41f3-82d7-959374abe111" volumeName="kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394416 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394423 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2becdfd-f3fd-4a40-8278-566d27fc1ebb" volumeName="kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394431 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08289af5-6136-4988-9455-9f2eef24caef" volumeName="kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394438 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8bf47b-05dc-4987-a7e9-7fa682c8ab60" volumeName="kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394445 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394452 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da5bf9f-3892-4b54-a337-880b1e19faf8" volumeName="kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394459 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="323fa83f-8f63-4a65-9837-d31e6f6de11d" volumeName="kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394467 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c729fe8-4a69-477a-a247-1e750d51c96a" volumeName="kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394473 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5cba4a-8119-475d-a365-f7977405ea6e" volumeName="kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394482 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08289af5-6136-4988-9455-9f2eef24caef" volumeName="kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394489 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26902984-9b1a-478b-8613-6ae684818d37" volumeName="kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394496 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7a69c4-438f-4679-9255-d0cae7ffc19f" volumeName="kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394503 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d" volumeName="kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394515 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e498e377-392b-421d-a455-1f7624968e6a" volumeName="kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394522 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e498e377-392b-421d-a455-1f7624968e6a" volumeName="kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394530 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394537 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306c0704-8d54-4fb6-a6f9-1f25bdc32289" volumeName="kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394544 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec41e273-2c37-467e-b55d-cef56c254d57" volumeName="kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394551 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec41e273-2c37-467e-b55d-cef56c254d57" volumeName="kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394559 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94567a77-bf25-464b-adde-cfb50ccfea99" volumeName="kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate" seLinuxMountContext="" Oct 30 21:13:39.394548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394566 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0217fb37-bb64-4dd6-aa3f-480891bd47dc" volumeName="kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394578 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1be52c02-d61a-4433-82d4-d4141294af25" volumeName="kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394596 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2da5bf9f-3892-4b54-a337-880b1e19faf8" volumeName="kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394604 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7a69c4-438f-4679-9255-d0cae7ffc19f" volumeName="kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394611 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585cbecc-cf75-4689-979e-3b70dd6cd0b3" volumeName="kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394618 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95e0daf-f36a-4951-8ead-9cf4ef55a928" volumeName="kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394625 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7a69c4-438f-4679-9255-d0cae7ffc19f" volumeName="kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394633 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f63d46-40a6-46c0-b359-d58509ccba89" volumeName="kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394639 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa" volumeName="kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394646 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a2ed00-d2f6-45cd-9085-7e93ce8031e1" volumeName="kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394653 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec41e273-2c37-467e-b55d-cef56c254d57" volumeName="kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394660 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274417e7-ffd3-4de2-a249-a15e27177905" volumeName="kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394667 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94567a77-bf25-464b-adde-cfb50ccfea99" volumeName="kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394677 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de5cba4a-8119-475d-a365-f7977405ea6e" volumeName="kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394685 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df904ba1-72fc-41f3-82d7-959374abe111" volumeName="kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk" seLinuxMountContext="" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394692 4918 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.394697 4918 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.395792 4918 factory.go:55] Registering systemd factory Oct 30 21:13:39.395956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.395807 4918 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396062 4918 factory.go:153] Registering CRI-O factory Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396084 4918 factory.go:223] Registration of the crio container factory successfully Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396164 4918 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396191 4918 factory.go:103] Registering Raw factory Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396219 4918 manager.go:1199] Started watching for new ooms in manager Oct 30 21:13:39.396751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.396689 4918 manager.go:319] Starting recovery of all containers Oct 30 21:13:39.400106 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.400086 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:39.400320 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.400299 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:39.405491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.405464 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:39.405697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.405681 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:39.418930 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.418898 4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:13:39.420104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.420085 4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:13:39.420104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.420103 4918 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:13:39.420248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.420119 4918 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:13:39.420248 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.420165 4918 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:13:39.427367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.427344 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:13:39.427753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.427730 4918 generic.go:356] "Generic (PLEG): container finished" podID="3b7a69c4-438f-4679-9255-d0cae7ffc19f" containerID="e00fc900a6f206e61fac07a2e08801ce79dd5f6ddf603bd9ded4d046c99f2354" exitCode=1 Oct 30 21:13:39.433455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.433436 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:39.433528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.433469 4918 generic.go:356] "Generic (PLEG): container finished" podID="de5cba4a-8119-475d-a365-f7977405ea6e" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" exitCode=255 Oct 30 21:13:39.449903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.449871 4918 generic.go:356] "Generic (PLEG): container finished" podID="0f2617ac5c29b7705758059c1bd8aae3" containerID="d1e8179fc42a69d86edd4ce0c7b4117dbc23779dbb0eb1feb15ed01673026fb4" exitCode=0 Oct 30 21:13:39.456483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456455 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="cd8f45ea957ed0faf9e11de5939230c3a29f851d87e5914f475f1baedb3aea2a" exitCode=0 Oct 30 21:13:39.456567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456483 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="715b2545af3aab8c1d219d04cffa6b0ba088ad83f730d7849a8c1b0c79292caa" exitCode=0 Oct 30 21:13:39.456567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456496 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="208c1e7de732f5a499108b9cc2698fca331bf3d099200683b807142a90e44572" exitCode=0 Oct 30 21:13:39.456567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456506 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="659efc90a9b01f24b4070576c59c8afdb408db372be1c7e50589968e1ae3520d" exitCode=0 Oct 30 21:13:39.456567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456515 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="2e97187cac90d51261f9e54adcaa594d8ed59ab18adf8b3499507c553525b344" exitCode=0 Oct 30 21:13:39.456567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.456527 4918 generic.go:356] "Generic (PLEG): container finished" podID="08289af5-6136-4988-9455-9f2eef24caef" containerID="7a7337b91ed5b4f84ec18ef7c52b3ae05ff15b5cb52793ad1f30c7d6b8e86e36" exitCode=0 Oct 30 21:13:39.485759 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.485744 4918 manager.go:324] Recovery completed Oct 30 21:13:39.491488 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.491469 4918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:13:39.501538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.501524 4918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:39.502548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.502533 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:39.502616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.502558 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:39.502616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.502569 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:39.505039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.504984 4918 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:13:39.505039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505005 4918 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:13:39.505039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505026 4918 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:13:39.505245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505227 4918 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:13:39.505292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505246 4918 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:13:39.505292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505270 4918 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:13:39.505292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505279 4918 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:13:39.505292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505287 4918 policy_none.go:50] "Start" Oct 30 21:13:39.505456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505307 4918 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:13:39.505456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.505320 4918 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:13:39.506208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.506184 4918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:39.507281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.507266 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:39.507342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.507292 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:39.507342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.507303 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:39.513715 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.513684 4918 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:13:39.513817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.513720 4918 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:13:39.513817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.513733 4918 policy_none.go:44] "Start" Oct 30 21:13:39.518117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.518094 4918 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:13:39.520642 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.520623 4918 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:13:39.520741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.520730 4918 manager.go:341] "Starting Device Plugin manager" Oct 30 21:13:39.520778 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.520761 4918 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:13:39.520778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.520769 4918 server.go:87] "Starting device plugin registration server" Oct 30 21:13:39.521019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.521007 4918 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:13:39.521079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.521021 4918 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:13:39.521126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.521108 4918 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:13:39.522160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.522119 4918 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:13:39.522236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.522174 4918 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:13:39.522236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.522182 4918 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:13:39.522342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.522261 4918 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:13:39.522156686Z","UUID":"04385add-24fe-4c7e-9012-fc2271631825","Handler":null,"Name":"","Endpoint":""} Oct 30 21:13:39.522674 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.522657 4918 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:13:39.522759 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.522696 4918 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-23.us-west-2.compute.internal\" not found" Oct 30 21:13:39.524043 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.524025 4918 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:13:39.524126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.524060 4918 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:13:39.567898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.567876 4918 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:39.568446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.568426 4918 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:39.568446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.568434 4918 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:39.615045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.615024 4918 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:39.615256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.615244 4918 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:13:39.621981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.621966 4918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:39.622814 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.622800 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:39.622862 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.622827 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:39.622862 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.622841 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-23.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:39.622929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.622883 4918 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.721702 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.721666 4918 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal"] Oct 30 21:13:39.722294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722263 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"76d614578e9abb3302e5026d743d87fe","Type":"ContainerStarted","Data":"89a3057ec0be59840fdc1e4618d421977c8c1b225f54533c2684a7d5707a51fa"} Oct 30 21:13:39.722354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722303 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"76d614578e9abb3302e5026d743d87fe","Type":"ContainerStarted","Data":"f946025773b7649ceb7fae87b499a44040eb8bf005a32afbf3b891955a6b0ccc"} Oct 30 21:13:39.722354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722335 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerStarted","Data":"45441e5e0180ce5d37ae2a951bc902e9768c9d0c2e62b3ee3e31c091126ecb3d"} Oct 30 21:13:39.722354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722350 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerDied","Data":"d1e8179fc42a69d86edd4ce0c7b4117dbc23779dbb0eb1feb15ed01673026fb4"} Oct 30 21:13:39.722514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722280 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.722514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722362 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" event={"ID":"0f2617ac5c29b7705758059c1bd8aae3","Type":"ContainerStarted","Data":"ac99841e42bd5b63f32c27dc4c7ee981ef109da27169b2fb89b0789b54d98c38"} Oct 30 21:13:39.722581 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.722481 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.725843 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.725829 4918 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.725919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.725907 4918 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.781800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.781758 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:39.781800 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.781793 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.782491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.782477 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:39.782529 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:39.782514 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.816015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.815992 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.816111 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.816019 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.816111 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.816080 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917107 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917132 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917151 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917185 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d614578e9abb3302e5026d743d87fe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"76d614578e9abb3302e5026d743d87fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917205 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:39.917292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:39.917216 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f2617ac5c29b7705758059c1bd8aae3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" (UID: \"0f2617ac5c29b7705758059c1bd8aae3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:40.369568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.369538 4918 apiserver.go:52] "Watching apiserver" Oct 30 21:13:40.474745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.474690 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:40.475321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.475296 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:40.513394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.513363 4918 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:13:40.514029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.514004 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq","kube-system/konnectivity-agent-c5d68","openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s","openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd","openshift-multus/multus-v2lxx","openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5","openshift-multus/network-metrics-daemon-kh6x7","kube-system/kas-connection-checker-685895c7c9-jz9n2","openshift-cluster-node-tuning-operator/tuned-dlx72","openshift-network-diagnostics/network-check-target-2qw92","kube-system/global-pull-secret-syncer-47nbz","openshift-ovn-kubernetes/ovnkube-node-gdshh","openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd","openshift-image-registry/node-ca-8nvnx","kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-sw2mt","kube-system/kas-connection-checker-685895c7c9-m4s2v","openshift-image-registry/image-registry-79dd8f4579-9l4tv","openshift-multus/multus-additional-cni-plugins-ng4gs","kube-system/kas-connection-checker-685895c7c9-c7qqx","openshift-network-console/networking-console-plugin-86c478bb88-nsh9q","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl","openshift-ingress-canary/ingress-canary-2cbsq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z","openshift-ingress/router-default-5b999d9c47-t55lr","openshift-image-registry/image-registry-5d7b77fdbb-tlfmz","openshift-dns/dns-default-7rk98","openshift-insights/insights-runtime-extractor-bn5sg","openshift-dns/node-resolver-5xv6n","openshift-console-operator/console-operator-7cd56b8dbc-crfvx","openshift-insights/insights-operator-7c4986d449-2q86m","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp"] Oct 30 21:13:40.515104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.515085 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.516968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.516948 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.517285 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.517269 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.517952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.517931 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.518279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.518263 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:40.519949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.519929 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:40.520886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.520870 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.526075 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.526056 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.526638 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.526621 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.532990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.532975 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:40.533056 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.533017 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:40.535861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.535848 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:40.535924 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.535879 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-23.us-west-2.compute.internal" Oct 30 21:13:40.540849 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.540817 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:40.566759 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566734 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.566854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566749 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.566854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566774 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.566854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566765 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:13:40.566854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566786 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566859 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566884 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566912 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566931 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.566981 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567009 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567030 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567032 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567052 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567059 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567093 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567115 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:13:40.567126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567125 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.567844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567226 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:13:40.567844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567341 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.567844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567364 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:13:40.567844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.567441 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.593367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.593339 4918 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:13:40.614104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614075 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:13:40.614212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614114 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:13:40.614212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614184 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:13:40.614354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614223 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:13:40.614354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614241 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:13:40.614354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614245 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:13:40.614354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614296 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.614354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614354 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614370 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614397 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614415 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614480 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614353 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614498 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614546 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614559 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614567 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w5d85\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614560 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614605 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614620 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614640 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.614637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614656 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h52wm\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614667 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614679 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614699 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4v9xp\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614701 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614743 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614759 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614796 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614800 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvf2h\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614830 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614835 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzjzq\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614875 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614890 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614904 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614921 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614930 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614935 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614960 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614980 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614984 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614989 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614988 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.614922 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615019 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615014 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615041 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615042 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dpfbq\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615079 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqpcg\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615084 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615101 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lrfkn\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615119 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615144 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615151 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615176 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fn5j\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615180 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615192 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615205 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph6zc\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615211 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.615393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615242 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-79sqh\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615473 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615489 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9zv66\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615497 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615499 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615519 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615515 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615534 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lm9pr\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615543 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615552 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615561 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q4s27\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615618 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwnd5\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615664 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615698 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615724 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9xqh\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615743 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615823 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:13:40.616476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.615856 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6zwr\"" Oct 30 21:13:40.622067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622024 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k4d\" (UniqueName: \"kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d\") pod \"kas-connection-checker-685895c7c9-c7qqx\" (UID: \"88f63d46-40a6-46c0-b359-d58509ccba89\") " pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:40.622067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622052 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.622148 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622068 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.622148 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622092 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.622148 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622136 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.622252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622163 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.622252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622188 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.622252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622213 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.622351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622264 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-tmp\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.622351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622268 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.622425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622405 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmgt\" (UniqueName: \"kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.622458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622438 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.622458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622411 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-script-lib\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.622523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622456 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.622523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622441 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95e0daf-f36a-4951-8ead-9cf4ef55a928-serving-cert\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.622523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622483 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.622616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622526 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.622616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622553 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.622689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622646 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.622689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622665 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w28n\" (UniqueName: \"kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n\") pod \"kas-connection-checker-685895c7c9-m4s2v\" (UID: \"f9771a55-0c02-4b30-9fe7-1850bd414cde\") " pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:40.622689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622682 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622702 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622721 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-service-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622746 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622754 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-tmp-dir\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622776 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622802 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622827 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622840 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-serviceca\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622866 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec41e273-2c37-467e-b55d-cef56c254d57-serving-cert\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.622881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622877 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622899 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622929 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622969 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.622977 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df904ba1-72fc-41f3-82d7-959374abe111-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623004 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623045 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623088 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-daemon-config\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623098 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de5cba4a-8119-475d-a365-f7977405ea6e-serving-cert\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623096 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-tuned\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623151 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.623182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623168 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585cbecc-cf75-4689-979e-3b70dd6cd0b3-tmp-dir\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623212 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623232 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623251 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623340 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623364 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623381 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623387 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/585cbecc-cf75-4689-979e-3b70dd6cd0b3-config-volume\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623426 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623446 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623507 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.623532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623522 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovn-node-metrics-cert\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623537 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623567 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-default-certificate\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623603 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbwr\" (UniqueName: \"kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623635 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623675 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623760 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623787 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623823 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623853 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623857 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-tmp\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623877 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623900 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623925 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623947 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.623987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623972 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.623996 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624006 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624022 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624034 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsdm\" (UniqueName: \"kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624036 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624052 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624066 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2blx\" (UniqueName: \"kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624138 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpv4q\" (UniqueName: \"kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624247 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624253 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624269 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e498e377-392b-421d-a455-1f7624968e6a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624344 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624432 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.624454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624450 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwdb\" (UniqueName: \"kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624483 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624505 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624601 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-config\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624608 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624681 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624739 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624757 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-env-overrides\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624760 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kn4c\" (UniqueName: \"kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624788 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624808 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.624844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624830 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624860 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr8r\" (UniqueName: \"kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624884 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624907 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624951 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.624991 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625004 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5bf9f-3892-4b54-a337-880b1e19faf8-agent-certs\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625030 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625071 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625092 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flq22\" (UniqueName: \"kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625132 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.625183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625168 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625186 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625218 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625244 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625349 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625382 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be52c02-d61a-4433-82d4-d4141294af25-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625390 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625409 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e95e0daf-f36a-4951-8ead-9cf4ef55a928-snapshots\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625423 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4k6\" (UniqueName: \"kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625459 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c729fe8-4a69-477a-a247-1e750d51c96a-iptables-alerter-script\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625526 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxb8f\" (UniqueName: \"kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625608 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.625622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625628 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625643 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625656 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5cba4a-8119-475d-a365-f7977405ea6e-trusted-ca\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625677 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6r5\" (UniqueName: \"kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625794 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625854 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz6j\" (UniqueName: \"kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j\") pod \"kas-connection-checker-685895c7c9-jz9n2\" (UID: \"323fa83f-8f63-4a65-9837-d31e6f6de11d\") " pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625896 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcfk\" (UniqueName: \"kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625926 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08289af5-6136-4988-9455-9f2eef24caef-cni-binary-copy\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625925 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.625996 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626024 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626063 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626086 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626104 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626122 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626138 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626206 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b7a69c4-438f-4679-9255-d0cae7ffc19f-ovnkube-config\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.626208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626216 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626232 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626246 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtgb\" (UniqueName: \"kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626273 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad304cde-8b80-44b3-8631-515bbd7b80fc-data-volume\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626279 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626291 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626308 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626337 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626410 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626452 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs2f\" (UniqueName: \"kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626479 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626503 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626519 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-stats-auth\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626538 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626565 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mcn\" (UniqueName: \"kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626592 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662r6\" (UniqueName: \"kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626632 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626684 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626751 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626773 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626798 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626815 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksvg\" (UniqueName: \"kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626831 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626842 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5bf9f-3892-4b54-a337-880b1e19faf8-konnectivity-ca\") pod \"konnectivity-agent-c5d68\" (UID: \"2da5bf9f-3892-4b54-a337-880b1e19faf8\") " pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626850 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626865 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df904ba1-72fc-41f3-82d7-959374abe111-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626876 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626898 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n259\" (UniqueName: \"kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.626971 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz72g\" (UniqueName: \"kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g\") pod \"migrator-66df5d496b-tbvbd\" (UID: \"d0bd86df-4448-46c9-aac0-2e8496b22533\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627025 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627046 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.627055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627058 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627062 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhph\" (UniqueName: \"kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph\") pod \"volume-data-source-validator-58c9f5f46d-6f4jl\" (UID: \"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627133 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627146 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/274417e7-ffd3-4de2-a249-a15e27177905-cert\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627161 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-original-pull-secret\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627197 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627215 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627216 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627238 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627328 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627350 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627359 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cni-binary-copy\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627392 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvrw\" (UniqueName: \"kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw\") pod \"network-check-source-6f78d59f99-gbtvd\" (UID: \"f2becdfd-f3fd-4a40-8278-566d27fc1ebb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627438 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627460 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627477 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627509 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627536 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627570 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627606 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627641 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627658 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.628463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.627693 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec41e273-2c37-467e-b55d-cef56c254d57-config\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.629415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.628834 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95e0daf-f36a-4951-8ead-9cf4ef55a928-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.681820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.681797 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpv4q\" (UniqueName: \"kubernetes.io/projected/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-kube-api-access-tpv4q\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.681820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.681804 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbwr\" (UniqueName: \"kubernetes.io/projected/ec41e273-2c37-467e-b55d-cef56c254d57-kube-api-access-tqbwr\") pod \"service-ca-operator-544d749c8c-rqm9s\" (UID: \"ec41e273-2c37-467e-b55d-cef56c254d57\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" Oct 30 21:13:40.681820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.681809 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsdm\" (UniqueName: \"kubernetes.io/projected/9c729fe8-4a69-477a-a247-1e750d51c96a-kube-api-access-nnsdm\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.682036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.682020 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwdb\" (UniqueName: \"kubernetes.io/projected/0217fb37-bb64-4dd6-aa3f-480891bd47dc-kube-api-access-chwdb\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.682070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.682020 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.682158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.682143 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2blx\" (UniqueName: \"kubernetes.io/projected/274417e7-ffd3-4de2-a249-a15e27177905-kube-api-access-h2blx\") pod \"ingress-canary-2cbsq\" (UID: \"274417e7-ffd3-4de2-a249-a15e27177905\") " pod="openshift-ingress-canary/ingress-canary-2cbsq" Oct 30 21:13:40.684334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.684312 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr8r\" (UniqueName: \"kubernetes.io/projected/dce12b75-7554-47a5-8214-f40f414060d7-kube-api-access-9xr8r\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:40.684484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.684466 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmgt\" (UniqueName: \"kubernetes.io/projected/94567a77-bf25-464b-adde-cfb50ccfea99-kube-api-access-hsmgt\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.684740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.684722 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6r5\" (UniqueName: \"kubernetes.io/projected/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-kube-api-access-fz6r5\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.684800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.684751 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w28n\" (UniqueName: \"kubernetes.io/projected/f9771a55-0c02-4b30-9fe7-1850bd414cde-kube-api-access-7w28n\") pod \"kas-connection-checker-685895c7c9-m4s2v\" (UID: \"f9771a55-0c02-4b30-9fe7-1850bd414cde\") " pod="kube-system/kas-connection-checker-685895c7c9-m4s2v" Oct 30 21:13:40.684834 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.684799 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flq22\" (UniqueName: \"kubernetes.io/projected/585cbecc-cf75-4689-979e-3b70dd6cd0b3-kube-api-access-flq22\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.685055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.685038 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k4d\" (UniqueName: \"kubernetes.io/projected/88f63d46-40a6-46c0-b359-d58509ccba89-kube-api-access-l2k4d\") pod \"kas-connection-checker-685895c7c9-c7qqx\" (UID: \"88f63d46-40a6-46c0-b359-d58509ccba89\") " pod="kube-system/kas-connection-checker-685895c7c9-c7qqx" Oct 30 21:13:40.685388 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.685367 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcfk\" (UniqueName: \"kubernetes.io/projected/df904ba1-72fc-41f3-82d7-959374abe111-kube-api-access-2pcfk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp\" (UID: \"df904ba1-72fc-41f3-82d7-959374abe111\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" Oct 30 21:13:40.685481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.685467 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.686633 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.686613 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662r6\" (UniqueName: \"kubernetes.io/projected/e95e0daf-f36a-4951-8ead-9cf4ef55a928-kube-api-access-662r6\") pod \"insights-operator-7c4986d449-2q86m\" (UID: \"e95e0daf-f36a-4951-8ead-9cf4ef55a928\") " pod="openshift-insights/insights-operator-7c4986d449-2q86m" Oct 30 21:13:40.686899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.686881 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4k6\" (UniqueName: \"kubernetes.io/projected/08289af5-6136-4988-9455-9f2eef24caef-kube-api-access-8c4k6\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.687074 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.687054 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs2f\" (UniqueName: \"kubernetes.io/projected/de5cba4a-8119-475d-a365-f7977405ea6e-kube-api-access-jxs2f\") pod \"console-operator-7cd56b8dbc-crfvx\" (UID: \"de5cba4a-8119-475d-a365-f7977405ea6e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:40.687152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.687075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz6j\" (UniqueName: \"kubernetes.io/projected/323fa83f-8f63-4a65-9837-d31e6f6de11d-kube-api-access-tsz6j\") pod \"kas-connection-checker-685895c7c9-jz9n2\" (UID: \"323fa83f-8f63-4a65-9837-d31e6f6de11d\") " pod="kube-system/kas-connection-checker-685895c7c9-jz9n2" Oct 30 21:13:40.687152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.687127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mcn\" (UniqueName: \"kubernetes.io/projected/3b7a69c4-438f-4679-9255-d0cae7ffc19f-kube-api-access-x9mcn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.687460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.687444 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtgb\" (UniqueName: \"kubernetes.io/projected/dfd123e3-d34f-411d-a77f-d147f5e3368e-kube-api-access-trtgb\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.689483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.689463 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz72g\" (UniqueName: \"kubernetes.io/projected/d0bd86df-4448-46c9-aac0-2e8496b22533-kube-api-access-bz72g\") pod \"migrator-66df5d496b-tbvbd\" (UID: \"d0bd86df-4448-46c9-aac0-2e8496b22533\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" Oct 30 21:13:40.689576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.689552 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxb8f\" (UniqueName: \"kubernetes.io/projected/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-kube-api-access-kxb8f\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.690299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.690279 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzxp\" (UniqueName: \"kubernetes.io/projected/06a00eb6-ceaa-49ee-8a17-01ba1009767e-kube-api-access-5mzxp\") pod \"network-check-target-2qw92\" (UID: \"06a00eb6-ceaa-49ee-8a17-01ba1009767e\") " pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:40.690340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.690279 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksvg\" (UniqueName: \"kubernetes.io/projected/e498e377-392b-421d-a455-1f7624968e6a-kube-api-access-6ksvg\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.690340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.690331 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhph\" (UniqueName: \"kubernetes.io/projected/e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa-kube-api-access-gdhph\") pod \"volume-data-source-validator-58c9f5f46d-6f4jl\" (UID: \"e6aa7aba-cd3c-4caf-8fec-df3b5aa043fa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6f4jl" Oct 30 21:13:40.695141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.695120 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n259\" (UniqueName: \"kubernetes.io/projected/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-api-access-4n259\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.695206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.695166 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kn4c\" (UniqueName: \"kubernetes.io/projected/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-kube-api-access-8kn4c\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.711580 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.711561 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.712173 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.712158 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvrw\" (UniqueName: \"kubernetes.io/projected/f2becdfd-f3fd-4a40-8278-566d27fc1ebb-kube-api-access-bwvrw\") pod \"network-check-source-6f78d59f99-gbtvd\" (UID: \"f2becdfd-f3fd-4a40-8278-566d27fc1ebb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-gbtvd" Oct 30 21:13:40.712866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.712852 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.728018 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.727998 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728023 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728039 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.728109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728054 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728073 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728126 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-system-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728140 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-kubelet-config\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728146 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728155 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-systemd\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728176 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-hostroot\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728181 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728197 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-lib-modules\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728203 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728226 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-netd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728232 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728257 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728273 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-var-lib-kubelet\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728276 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728295 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728303 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-conf-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728313 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728319 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728332 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728336 4918 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728347 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-cni-bin\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728347 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728378 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728390 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.228375352 +0000 UTC m=+1.921857595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728406 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-cnibin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728414 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728416 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-netns\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728433 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728452 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728464 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-system-cni-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728469 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.228454422 +0000 UTC m=+1.921936656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : secret "metrics-daemon-secret" not found Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728484 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-cni-dir\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728502 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728525 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728549 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.728570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728559 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-modprobe-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728587 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-var-lib-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728607 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728612 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728624 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728636 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728645 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.228633585 +0000 UTC m=+1.922115806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728667 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-netns\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728678 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.228663685 +0000 UTC m=+1.922145913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728682 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-multus-socket-dir-parent\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728687 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728668 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728734 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728753 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728768 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728770 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71cd5ea6-5cf5-46b7-a778-7a2921fbd837-host\") pod \"node-ca-8nvnx\" (UID: \"71cd5ea6-5cf5-46b7-a778-7a2921fbd837\") " pod="openshift-image-registry/node-ca-8nvnx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728788 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-kubelet\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728797 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728820 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728835 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728849 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728861 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-registration-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728794 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-ovn\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728863 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728892 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-k8s-cni-cncf-io\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728878 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-run-systemd\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728903 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728928 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-device-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728943 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728952 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728904 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c729fe8-4a69-477a-a247-1e750d51c96a-host-slash\") pod \"iptables-alerter-sw2mt\" (UID: \"9c729fe8-4a69-477a-a247-1e750d51c96a\") " pod="openshift-network-operator/iptables-alerter-sw2mt" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.728982 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.228969818 +0000 UTC m=+1.922452059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.728996 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-etc-kubernetes\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729008 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729034 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729043 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-d\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729058 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-systemd-units\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729059 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729078 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-etc-openvswitch\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729081 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729099 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-slash\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729106 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729122 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729143 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729156 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysconfig\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729163 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729172 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729178 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729190 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729205 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729210 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-bin\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729209 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-socket-dir\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729217 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729161 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-run-multus-certs\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729236 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.229226024 +0000 UTC m=+1.922708241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729257 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-host\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729263 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729276 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-run\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729278 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7cd2ff9-7aeb-4ebf-9605-a213958c1153-dbus\") pod \"global-pull-secret-syncer-47nbz\" (UID: \"d7cd2ff9-7aeb-4ebf-9605-a213958c1153\") " pod="kube-system/global-pull-secret-syncer-47nbz" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729290 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfd123e3-d34f-411d-a77f-d147f5e3368e-sys-fs\") pod \"aws-ebs-csi-driver-node-2v52z\" (UID: \"dfd123e3-d34f-411d-a77f-d147f5e3368e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v52z" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729290 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729330 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729346 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729352 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729369 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-cnibin\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.729344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729379 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729390 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-cni-multus\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729407 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.229397911 +0000 UTC m=+1.922880132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729419 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d-hosts-file\") pod \"node-resolver-5xv6n\" (UID: \"d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d\") " pod="openshift-dns/node-resolver-5xv6n" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729421 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729433 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8bf47b-05dc-4987-a7e9-7fa682c8ab60-host-var-lib-kubelet\") pod \"multus-v2lxx\" (UID: \"1d8bf47b-05dc-4987-a7e9-7fa682c8ab60\") " pod="openshift-multus/multus-v2lxx" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729445 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-node-log\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729459 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729480 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729486 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729516 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.22950479 +0000 UTC m=+1.922987011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729510 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729537 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729541 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729567 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729576 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729583 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.229573886 +0000 UTC m=+1.923056104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729622 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.229613613 +0000 UTC m=+1.923095856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729630 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-log-socket\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729641 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729656 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729672 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729700 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729730 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-sys\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729743 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729768 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729772 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b7a69c4-438f-4679-9255-d0cae7ffc19f-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdshh\" (UID: \"3b7a69c4-438f-4679-9255-d0cae7ffc19f\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729782 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad304cde-8b80-44b3-8631-515bbd7b80fc-crio-socket\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729744 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08289af5-6136-4988-9455-9f2eef24caef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ng4gs\" (UID: \"08289af5-6136-4988-9455-9f2eef24caef\") " pod="openshift-multus/multus-additional-cni-plugins-ng4gs" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729793 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729794 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-sysctl-conf\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.729818 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a2ed00-d2f6-45cd-9085-7e93ce8031e1-etc-kubernetes\") pod \"tuned-dlx72\" (UID: \"e9a2ed00-d2f6-45cd-9085-7e93ce8031e1\") " pod="openshift-cluster-node-tuning-operator/tuned-dlx72" Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729872 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:40.731201 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:40.729915 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:41.229906028 +0000 UTC m=+1.923388245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:40.816923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.816901 4918 scope.go:122] "RemoveContainer" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" Oct 30 21:13:40.820958 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.820942 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:40.828325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:40.828309 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:13:41.235133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235065 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:41.235133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235099 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:41.235133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235131 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235151 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235192 4918 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235216 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235230 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235239 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235249 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235230774 +0000 UTC m=+2.928712992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : configmap "kube-rbac-proxy" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235308 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235290146 +0000 UTC m=+2.928772360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : secret "metrics-daemon-secret" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235332 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235315031 +0000 UTC m=+2.928797244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:41.235398 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235399 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235408 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235442 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235422612 +0000 UTC m=+2.928904825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235466 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235459133 +0000 UTC m=+2.928941346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235488 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235499 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235502 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235529 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235537 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235526054 +0000 UTC m=+2.929008275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235566 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235575 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235603 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235608 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235608 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235598854 +0000 UTC m=+2.929081097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235643 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235654 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235662 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235651048 +0000 UTC m=+2.929133261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235680 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235670037 +0000 UTC m=+2.929152257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235696 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235688528 +0000 UTC m=+2.929170742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.235774 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235865 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:41.235886 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.235894 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.235887413 +0000 UTC m=+2.929369631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:41.477813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.477787 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:41.479444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.479426 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/2.log" Oct 30 21:13:41.479873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.479859 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/1.log" Oct 30 21:13:41.479943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.479888 4918 generic.go:356] "Generic (PLEG): container finished" podID="de5cba4a-8119-475d-a365-f7977405ea6e" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" exitCode=255 Oct 30 21:13:41.479943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.479928 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerDied","Data":"50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d"} Oct 30 21:13:41.480019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.479980 4918 scope.go:122] "RemoveContainer" containerID="8beeea0610ba930156a0b2cee7d9d0759f8170c20b9407ef00e74a15ca9ff27a" Oct 30 21:13:41.480470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.480452 4918 scope.go:122] "RemoveContainer" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" Oct 30 21:13:41.480623 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:41.480605 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:41.909518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:41.909494 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:42.245957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.245865 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:42.245957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.245910 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:42.245957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.245947 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.245970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.245999 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246022 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246036 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246054 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246075 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246107 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246088423 +0000 UTC m=+4.939570655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : secret "metrics-daemon-secret" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246111 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246134 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246152 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246140629 +0000 UTC m=+4.939622857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246155 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246184 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246196 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls podName:ad304cde-8b80-44b3-8631-515bbd7b80fc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246183231 +0000 UTC m=+4.939665460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bn5sg" (UID: "ad304cde-8b80-44b3-8631-515bbd7b80fc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246236 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246243 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:42.246272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246253 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246307 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246318 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246322 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246369 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246358719 +0000 UTC m=+4.939840950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246386 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246389 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls podName:585cbecc-cf75-4689-979e-3b70dd6cd0b3 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246379599 +0000 UTC m=+4.939861865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls") pod "dns-default-7rk98" (UID: "585cbecc-cf75-4689-979e-3b70dd6cd0b3") : secret "dns-default-metrics-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246406 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.24639838 +0000 UTC m=+4.939880599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246421 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246413856 +0000 UTC m=+4.939896076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246436 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert podName:1be52c02-d61a-4433-82d4-d4141294af25 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246429256 +0000 UTC m=+4.939911482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsh9q" (UID: "1be52c02-d61a-4433-82d4-d4141294af25") : secret "networking-console-plugin-cert" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246445 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad304cde-8b80-44b3-8631-515bbd7b80fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246472 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246448868 +0000 UTC m=+4.939931082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.246531 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246637 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:42.247067 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:42.246697 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:44.246682939 +0000 UTC m=+4.940165169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:42.377918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.377877 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:42.417873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.417847 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:42.485555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.485535 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/2.log" Oct 30 21:13:42.485953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.485700 4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:13:42.501344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.501293 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5xv6n_d22b5a4e-5e79-4380-b6d2-2d7eb3e5b04d/dns-node-resolver/0.log" Oct 30 21:13:42.936031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:42.936007 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:13:43.007302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.007275 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:43.194360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.194303 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:43.457021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.456949 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:43.490389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.490357 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbvbd" event={"ID":"d0bd86df-4448-46c9-aac0-2e8496b22533","Type":"ContainerStarted","Data":"673f3e243e989749188717e0628a56e28c625aabc8f3385977070b307e93f748"} Oct 30 21:13:43.581677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.581647 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:43.581943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.581931 4918 scope.go:122] "RemoveContainer" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" Oct 30 21:13:43.582081 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:43.582064 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:43.699052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.699028 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8nvnx_71cd5ea6-5cf5-46b7-a778-7a2921fbd837/node-ca/0.log" Oct 30 21:13:43.884722 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:43.884686 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.260642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.260581 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.263654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263633 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:44.263790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263671 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:44.263790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263736 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:44.263790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263770 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263791 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263810 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263845 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.263829394 +0000 UTC m=+8.957311643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263871 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263895 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263903 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:44.263951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263931 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.263974 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263909 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264036 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263909 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264054 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.264041 4918 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264068 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.263988 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.263973587 +0000 UTC m=+8.957455822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : configmap references non-existent config key: service-ca.crt Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264122 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.264110482 +0000 UTC m=+8.957592700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264140 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.264130846 +0000 UTC m=+8.957613061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264158 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.264149291 +0000 UTC m=+8.957631511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264173 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.264165674 +0000 UTC m=+8.957647892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:44.264312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.264220 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:44.265008 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264350 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:44.265008 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:44.264406 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs podName:dce12b75-7554-47a5-8214-f40f414060d7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:48.264393138 +0000 UTC m=+8.957875356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs") pod "network-metrics-daemon-kh6x7" (UID: "dce12b75-7554-47a5-8214-f40f414060d7") : secret "metrics-daemon-secret" not found Oct 30 21:13:44.266405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.266376 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad304cde-8b80-44b3-8631-515bbd7b80fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bn5sg\" (UID: \"ad304cde-8b80-44b3-8631-515bbd7b80fc\") " pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:44.266405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.266386 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/585cbecc-cf75-4689-979e-3b70dd6cd0b3-metrics-tls\") pod \"dns-default-7rk98\" (UID: \"585cbecc-cf75-4689-979e-3b70dd6cd0b3\") " pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:44.266555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.266468 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be52c02-d61a-4433-82d4-d4141294af25-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsh9q\" (UID: \"1be52c02-d61a-4433-82d4-d4141294af25\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:44.299827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.299789 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2cbsq_274417e7-ffd3-4de2-a249-a15e27177905/serve-healthcheck-canary/0.log" Oct 30 21:13:44.418697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.418674 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" Oct 30 21:13:44.426598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.426574 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:44.427686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.427668 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bn5sg" Oct 30 21:13:44.662035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.661995 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.788033 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.788009 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsh9q"] Oct 30 21:13:44.825420 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:44.825379 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be52c02_d61a_4433_82d4_d4141294af25.slice/crio-7f33e71808abdb2c16863bc90866bc880856ef2f2b4ea9c9d10f73cfba2040b6 WatchSource:0}: Error finding container 7f33e71808abdb2c16863bc90866bc880856ef2f2b4ea9c9d10f73cfba2040b6: Status 404 returned error can't find the container with id 7f33e71808abdb2c16863bc90866bc880856ef2f2b4ea9c9d10f73cfba2040b6 Oct 30 21:13:44.839401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.839383 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rk98"] Oct 30 21:13:44.842184 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:44.842162 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585cbecc_cf75_4689_979e_3b70dd6cd0b3.slice/crio-7bd9eafc02f3d0ce255a42b80e806ac91e80f6690df8db0705bc61976c61a456 WatchSource:0}: Error finding container 7bd9eafc02f3d0ce255a42b80e806ac91e80f6690df8db0705bc61976c61a456: Status 404 returned error can't find the container with id 7bd9eafc02f3d0ce255a42b80e806ac91e80f6690df8db0705bc61976c61a456 Oct 30 21:13:44.891240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.891214 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bn5sg"] Oct 30 21:13:44.894487 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:44.894469 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad304cde_8b80_44b3_8631_515bbd7b80fc.slice/crio-9476a4e44047f73e38e3fb5080ba35bd6932585a4e372fd083da4b6b4853d26d WatchSource:0}: Error finding container 9476a4e44047f73e38e3fb5080ba35bd6932585a4e372fd083da4b6b4853d26d: Status 404 returned error can't find the container with id 9476a4e44047f73e38e3fb5080ba35bd6932585a4e372fd083da4b6b4853d26d Oct 30 21:13:44.913822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.913775 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.974569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:44.974549 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tbvbd_d0bd86df-4448-46c9-aac0-2e8496b22533/migrator/0.log" Oct 30 21:13:45.104189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.104165 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp_df904ba1-72fc-41f3-82d7-959374abe111/kube-storage-version-migrator-operator/0.log" Oct 30 21:13:45.108717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.108677 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:45.456167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.456129 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:45.499661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.499636 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" event={"ID":"1be52c02-d61a-4433-82d4-d4141294af25","Type":"ContainerStarted","Data":"7f33e71808abdb2c16863bc90866bc880856ef2f2b4ea9c9d10f73cfba2040b6"} Oct 30 21:13:45.501005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.500982 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bn5sg" event={"ID":"ad304cde-8b80-44b3-8631-515bbd7b80fc","Type":"ContainerStarted","Data":"f26d8078674432285c15591289a0cffe8e9bba0fa11ef4530a865ac6121c8666"} Oct 30 21:13:45.501085 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.501030 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bn5sg" event={"ID":"ad304cde-8b80-44b3-8631-515bbd7b80fc","Type":"ContainerStarted","Data":"9476a4e44047f73e38e3fb5080ba35bd6932585a4e372fd083da4b6b4853d26d"} Oct 30 21:13:45.502054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.502032 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rk98" event={"ID":"585cbecc-cf75-4689-979e-3b70dd6cd0b3","Type":"ContainerStarted","Data":"7bd9eafc02f3d0ce255a42b80e806ac91e80f6690df8db0705bc61976c61a456"} Oct 30 21:13:45.721220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:45.721144 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdshh" Oct 30 21:13:46.262723 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:46.262670 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.657476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:46.657447 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.861010 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:46.860954 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:47.056444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.056416 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:47.458920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.458851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:47.511070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.511042 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bn5sg" event={"ID":"ad304cde-8b80-44b3-8631-515bbd7b80fc","Type":"ContainerStarted","Data":"89270381b5943a11a2c59d56693940ddf2434095a0af80ee93f2d956e0e186ce"} Oct 30 21:13:47.512820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.512792 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rk98" event={"ID":"585cbecc-cf75-4689-979e-3b70dd6cd0b3","Type":"ContainerStarted","Data":"304b919b70c1063b6d9bbdb7ad7f3dedf10c3c81543a3c93e69748e2fc8a6beb"} Oct 30 21:13:47.512945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.512840 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rk98" event={"ID":"585cbecc-cf75-4689-979e-3b70dd6cd0b3","Type":"ContainerStarted","Data":"d6926bbc6c310679b43d0aaf3a77ed307831aa9691d3588624546f143c54cdf6"} Oct 30 21:13:47.512945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.512892 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:47.514222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.514205 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsh9q" event={"ID":"1be52c02-d61a-4433-82d4-d4141294af25","Type":"ContainerStarted","Data":"72d9cc6abfe6010ccd47b619bd147a29a8a058f994e0111a85e67dac37536f03"} Oct 30 21:13:47.650403 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:47.650370 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.253202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.253171 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.296325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296292 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:48.296484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296350 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:48.296484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:48.296484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296442 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:48.296484 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296458 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:48.296484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296483 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296526 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls podName:e498e377-392b-421d-a455-1f7624968e6a nodeName:}" failed. No retries permitted until 2026-10-30 21:13:56.296508201 +0000 UTC m=+16.989990417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-thvb5" (UID: "e498e377-392b-421d-a455-1f7624968e6a") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296584 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296648 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls podName:0217fb37-bb64-4dd6-aa3f-480891bd47dc nodeName:}" failed. No retries permitted until 2026-10-30 21:13:56.296626848 +0000 UTC m=+16.990109097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zmjtq" (UID: "0217fb37-bb64-4dd6-aa3f-480891bd47dc") : secret "samples-operator-tls" not found Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296648 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296687 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs podName:94567a77-bf25-464b-adde-cfb50ccfea99 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:56.296677505 +0000 UTC m=+16.990159719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs") pod "router-default-5b999d9c47-t55lr" (UID: "94567a77-bf25-464b-adde-cfb50ccfea99") : secret "router-metrics-certs-default" not found Oct 30 21:13:48.296770 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296767 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:48.297018 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296781 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b77fdbb-tlfmz: secret "image-registry-tls" not found Oct 30 21:13:48.297018 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296816 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:48.297018 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.296840 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls podName:26902984-9b1a-478b-8613-6ae684818d37 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:56.296823866 +0000 UTC m=+16.990306095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls") pod "image-registry-5d7b77fdbb-tlfmz" (UID: "26902984-9b1a-478b-8613-6ae684818d37") : secret "image-registry-tls" not found Oct 30 21:13:48.297018 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.296931 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:48.297171 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.297045 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:13:48.297171 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.297056 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79dd8f4579-9l4tv: secret "image-registry-tls" not found Oct 30 21:13:48.297171 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:48.297089 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls podName:306c0704-8d54-4fb6-a6f9-1f25bdc32289 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:56.297077943 +0000 UTC m=+16.990560171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls") pod "image-registry-79dd8f4579-9l4tv" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289") : secret "image-registry-tls" not found Oct 30 21:13:48.297471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.297451 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94567a77-bf25-464b-adde-cfb50ccfea99-service-ca-bundle\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:48.298941 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.298910 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce12b75-7554-47a5-8214-f40f414060d7-metrics-certs\") pod \"network-metrics-daemon-kh6x7\" (UID: \"dce12b75-7554-47a5-8214-f40f414060d7\") " pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:48.321062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.321039 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kh6x7" Oct 30 21:13:48.454041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.454005 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.689700 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.689542 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kh6x7"] Oct 30 21:13:48.693358 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:48.693335 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce12b75_7554_47a5_8214_f40f414060d7.slice/crio-2ebf054e4200424469c65041527b1624836912dd5046947a15d1351cf009cb4f WatchSource:0}: Error finding container 2ebf054e4200424469c65041527b1624836912dd5046947a15d1351cf009cb4f: Status 404 returned error can't find the container with id 2ebf054e4200424469c65041527b1624836912dd5046947a15d1351cf009cb4f Oct 30 21:13:48.855602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:48.855547 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.053211 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.053178 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.522239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.522205 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kh6x7" event={"ID":"dce12b75-7554-47a5-8214-f40f414060d7","Type":"ContainerStarted","Data":"2ebf054e4200424469c65041527b1624836912dd5046947a15d1351cf009cb4f"} Oct 30 21:13:49.524288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.524262 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bn5sg" event={"ID":"ad304cde-8b80-44b3-8631-515bbd7b80fc","Type":"ContainerStarted","Data":"fa894023f8742c2102d6171d05da5276f4419ed0be0ad35769f8817c71ff455a"} Oct 30 21:13:49.676125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.676092 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.739618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.739588 4918 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:49.739968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:49.739955 4918 scope.go:122] "RemoveContainer" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" Oct 30 21:13:49.740134 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:49.740118 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:50.034812 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.034766 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.269931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.269895 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.457483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.457431 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.529628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.529600 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kh6x7" event={"ID":"dce12b75-7554-47a5-8214-f40f414060d7","Type":"ContainerStarted","Data":"00f08bd06a2ae47bb597fb0f238f43b4be86b36ca36872e2700cd3a28e94972f"} Oct 30 21:13:50.529989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.529654 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kh6x7" event={"ID":"dce12b75-7554-47a5-8214-f40f414060d7","Type":"ContainerStarted","Data":"df5ebad11921dc4c33178071fabb45917b930ed8c7f07984c07f1cb1286c52fe"} Oct 30 21:13:50.643322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.643299 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.863484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:50.863458 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:51.050184 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.050117 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:51.236618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.236585 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:51.420689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.420667 4918 scope.go:122] "RemoveContainer" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" Oct 30 21:13:51.535612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.535594 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:13:51.536045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.536029 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/2.log" Oct 30 21:13:51.536178 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.536066 4918 generic.go:356] "Generic (PLEG): container finished" podID="de5cba4a-8119-475d-a365-f7977405ea6e" containerID="31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff" exitCode=255 Oct 30 21:13:51.536178 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.536137 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerDied","Data":"31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff"} Oct 30 21:13:51.536289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.536189 4918 scope.go:122] "RemoveContainer" containerID="50ac04bfa4c0c85fb3a61713beebb2b89fff872344b3af7e3ab18570c60b564d" Oct 30 21:13:51.536803 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.536780 4918 scope.go:122] "RemoveContainer" containerID="31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff" Oct 30 21:13:51.537064 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:51.537043 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:51.654022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:51.653984 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.247630 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:52.247596 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.477844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:52.477813 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.542746 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:52.542671 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:13:52.853486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:52.853427 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.054499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:53.054470 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.255472 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:53.255443 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.582720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:53.582635 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:53.583109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:53.582957 4918 scope.go:122] "RemoveContainer" containerID="31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff" Oct 30 21:13:53.583160 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:53.583129 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:13:54.941736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:54.941691 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.119368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:55.119337 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:56.010479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.010424 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:56.207998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.207966 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:56.361822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.361799 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:56.361953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.361843 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:56.361953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.361882 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:56.361953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.361904 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:56.361953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.361930 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:56.364765 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.364736 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e498e377-392b-421d-a455-1f7624968e6a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-thvb5\" (UID: \"e498e377-392b-421d-a455-1f7624968e6a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:56.364901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.364778 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0217fb37-bb64-4dd6-aa3f-480891bd47dc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zmjtq\" (UID: \"0217fb37-bb64-4dd6-aa3f-480891bd47dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:56.364901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.364854 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"image-registry-79dd8f4579-9l4tv\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:56.365065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.365044 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94567a77-bf25-464b-adde-cfb50ccfea99-metrics-certs\") pod \"router-default-5b999d9c47-t55lr\" (UID: \"94567a77-bf25-464b-adde-cfb50ccfea99\") " pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:56.365175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.365154 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"image-registry-5d7b77fdbb-tlfmz\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:56.415789 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.415761 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" Oct 30 21:13:56.417939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.417920 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:56.418031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.418012 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:56.419077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.419056 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" Oct 30 21:13:56.421540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.421511 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:56.835401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.835337 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5"] Oct 30 21:13:56.838327 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:56.838306 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode498e377_392b_421d_a455_1f7624968e6a.slice/crio-f238f143282967a8bea937a332ddb57ea92f1bab5dbd78c40952d8e7915210e3 WatchSource:0}: Error finding container f238f143282967a8bea937a332ddb57ea92f1bab5dbd78c40952d8e7915210e3: Status 404 returned error can't find the container with id f238f143282967a8bea937a332ddb57ea92f1bab5dbd78c40952d8e7915210e3 Oct 30 21:13:56.885933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.883838 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:13:56.885933 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:56.884118 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26902984_9b1a_478b_8613_6ae684818d37.slice/crio-e6aa6a5361e05c1f3b7d189ceb0e44ee755b07e33eadf5349e1d7156be635eb1 WatchSource:0}: Error finding container e6aa6a5361e05c1f3b7d189ceb0e44ee755b07e33eadf5349e1d7156be635eb1: Status 404 returned error can't find the container with id e6aa6a5361e05c1f3b7d189ceb0e44ee755b07e33eadf5349e1d7156be635eb1 Oct 30 21:13:56.900534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.900510 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq"] Oct 30 21:13:56.909683 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:13:56.909659 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94567a77_bf25_464b_adde_cfb50ccfea99.slice/crio-1ed4c82b9b002cd57ac692cf7a597adfa67549888105b7896c0da8d6ff8f6c13 WatchSource:0}: Error finding container 1ed4c82b9b002cd57ac692cf7a597adfa67549888105b7896c0da8d6ff8f6c13: Status 404 returned error can't find the container with id 1ed4c82b9b002cd57ac692cf7a597adfa67549888105b7896c0da8d6ff8f6c13 Oct 30 21:13:56.928359 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.928337 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b999d9c47-t55lr"] Oct 30 21:13:56.939269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:56.939249 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:13:57.520954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.520928 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rk98" Oct 30 21:13:57.560437 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.560400 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" event={"ID":"e498e377-392b-421d-a455-1f7624968e6a","Type":"ContainerStarted","Data":"f238f143282967a8bea937a332ddb57ea92f1bab5dbd78c40952d8e7915210e3"} Oct 30 21:13:57.561795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.561765 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" event={"ID":"26902984-9b1a-478b-8613-6ae684818d37","Type":"ContainerStarted","Data":"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019"} Oct 30 21:13:57.561853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.561801 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" event={"ID":"26902984-9b1a-478b-8613-6ae684818d37","Type":"ContainerStarted","Data":"e6aa6a5361e05c1f3b7d189ceb0e44ee755b07e33eadf5349e1d7156be635eb1"} Oct 30 21:13:57.561853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.561827 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:13:57.563470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.563446 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" event={"ID":"306c0704-8d54-4fb6-a6f9-1f25bdc32289","Type":"ContainerStarted","Data":"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b"} Oct 30 21:13:57.563557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.563487 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" event={"ID":"306c0704-8d54-4fb6-a6f9-1f25bdc32289","Type":"ContainerStarted","Data":"4d16429807999ff39fdddde0cfadad1da13a4a7be1f88055615b88ed57c85d45"} Oct 30 21:13:57.563557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.563524 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:13:57.564926 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.564907 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b999d9c47-t55lr" event={"ID":"94567a77-bf25-464b-adde-cfb50ccfea99","Type":"ContainerStarted","Data":"a157fd64f428862729a2c2bc0712fc9b5bd016f1d8c3ff6a26da53cb1509a7b2"} Oct 30 21:13:57.565017 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.564943 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b999d9c47-t55lr" event={"ID":"94567a77-bf25-464b-adde-cfb50ccfea99","Type":"ContainerStarted","Data":"1ed4c82b9b002cd57ac692cf7a597adfa67549888105b7896c0da8d6ff8f6c13"} Oct 30 21:13:57.566039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.566016 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" event={"ID":"0217fb37-bb64-4dd6-aa3f-480891bd47dc","Type":"ContainerStarted","Data":"583f93580597b2fd16f4546919bde03426eea4db510ca1dcb19f7bf28d5df834"} Oct 30 21:13:57.615662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.615640 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:13:57.962176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:57.962144 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:58.149570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.149530 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:58.352206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.352126 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:58.418783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.418754 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:58.422049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.422024 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:58.563634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.563600 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:58.570598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.570566 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:58.571930 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:58.571906 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b999d9c47-t55lr" Oct 30 21:13:59.575442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:59.575362 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" event={"ID":"0217fb37-bb64-4dd6-aa3f-480891bd47dc","Type":"ContainerStarted","Data":"0f155d6e6e0b03c88b6f782cb8afac15b17ab3f1bbd6cf4a50d60575a4fbbea5"} Oct 30 21:13:59.575442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:59.575417 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zmjtq" event={"ID":"0217fb37-bb64-4dd6-aa3f-480891bd47dc","Type":"ContainerStarted","Data":"e983f0c4580499747de477758cf1f51808125be3bd4798fbbd78c33210590d1f"} Oct 30 21:13:59.577165 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:59.577112 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-thvb5" event={"ID":"e498e377-392b-421d-a455-1f7624968e6a","Type":"ContainerStarted","Data":"81ab0412767e1a65c9d5bf49df47ad3c1c2e928f09e0450f95188ab684b1586f"} Oct 30 21:13:59.740356 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:59.740328 4918 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:13:59.740752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:13:59.740733 4918 scope.go:122] "RemoveContainer" containerID="31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff" Oct 30 21:13:59.740955 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:13:59.740933 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-crfvx_openshift-console-operator(de5cba4a-8119-475d-a365-f7977405ea6e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" Oct 30 21:14:00.411564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:00.411528 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.607830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:00.607802 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.811064 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:00.811003 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:01.529641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:01.529610 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:01.749677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:01.749641 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:03.480903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:03.480871 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:03.730177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:03.730141 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:03.907229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:03.907201 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.802917 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:05.802880 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.989215 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:05.989188 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.184001 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:06.183976 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.369664 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:06.369640 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:07.236986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:07.236958 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5d68" Oct 30 21:14:09.942840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:09.942801 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:10.129141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:10.129110 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:11.193977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:11.193936 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:11.978045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:11.978017 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.249690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.249608 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.494613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.494555 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.698801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.698772 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.785444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.785414 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ghxxm"] Oct 30 21:14:12.834152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.834128 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.889734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.889681 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.910311 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.910285 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:14:12.930485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.930463 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xhltr\"" Oct 30 21:14:12.930485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.930478 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:14:12.938448 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.938425 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2qw92" Oct 30 21:14:12.956425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.956372 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:14:12.956425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.956393 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:14:12.979441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979416 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-textfile\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979443 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-tls\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979474 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-sys\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979525 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979632 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979611 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979673 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979659 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-wtmp\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979724 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-metrics-client-ca\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979758 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/33b37892-bcc7-422f-a15d-71b0abeed209-kube-api-access-jrhjj\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:12.979816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:12.979779 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-root\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080207 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-textfile\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080239 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-tls\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080443 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080421 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-sys\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080494 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080521 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-sys\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080530 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-wtmp\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080577 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-metrics-client-ca\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080607 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/33b37892-bcc7-422f-a15d-71b0abeed209-kube-api-access-jrhjj\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080628 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-textfile\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080653 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-wtmp\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080666 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-root\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.080882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.080635 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b37892-bcc7-422f-a15d-71b0abeed209-root\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.081231 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.081203 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.081231 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.081224 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b37892-bcc7-422f-a15d-71b0abeed209-metrics-client-ca\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.083069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.083047 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-tls\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.083641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.083621 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.084203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.084185 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b37892-bcc7-422f-a15d-71b0abeed209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.138014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.137992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhjj\" (UniqueName: \"kubernetes.io/projected/33b37892-bcc7-422f-a15d-71b0abeed209-kube-api-access-jrhjj\") pod \"node-exporter-ghxxm\" (UID: \"33b37892-bcc7-422f-a15d-71b0abeed209\") " pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.188954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.188933 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghxxm" Oct 30 21:14:13.220208 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:14:13.220181 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b37892_bcc7_422f_a15d_71b0abeed209.slice/crio-b5f3ac29d73922d0ce920ccd0991ee66b4e8c2fb6b83bb4473444c1f6aa83981 WatchSource:0}: Error finding container b5f3ac29d73922d0ce920ccd0991ee66b4e8c2fb6b83bb4473444c1f6aa83981: Status 404 returned error can't find the container with id b5f3ac29d73922d0ce920ccd0991ee66b4e8c2fb6b83bb4473444c1f6aa83981 Oct 30 21:14:13.277695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.277671 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.421228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.421201 4918 scope.go:122] "RemoveContainer" containerID="31ea95c918b85a12595889d925e2f170c0345db24ebd14677986e9edbbceacff" Oct 30 21:14:13.567951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.567926 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.624276 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.624254 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:14:13.624398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.624368 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" event={"ID":"de5cba4a-8119-475d-a365-f7977405ea6e","Type":"ContainerStarted","Data":"82b538c273512d7f88572bd1122b4b48dcdbb80db9e9e39f78779283aa3b94dd"} Oct 30 21:14:13.624665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.624643 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:14:13.625637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.625611 4918 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-crfvx container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Oct 30 21:14:13.625743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.625672 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" podUID="de5cba4a-8119-475d-a365-f7977405ea6e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" Oct 30 21:14:13.625800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.625746 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghxxm" event={"ID":"33b37892-bcc7-422f-a15d-71b0abeed209","Type":"ContainerStarted","Data":"b5f3ac29d73922d0ce920ccd0991ee66b4e8c2fb6b83bb4473444c1f6aa83981"} Oct 30 21:14:13.754432 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:13.754328 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.165498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:14.165468 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.469642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:14.469543 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.637387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:14.637359 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-crfvx" Oct 30 21:14:15.118184 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.118152 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:15.423860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.423784 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:15.637757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.637697 4918 generic.go:356] "Generic (PLEG): container finished" podID="33b37892-bcc7-422f-a15d-71b0abeed209" containerID="bb65cd926393867221d9f8e0d3f44e41905476a926fd4e91f9e8c300eb09f0ba" exitCode=0 Oct 30 21:14:15.638198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.637779 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghxxm" event={"ID":"33b37892-bcc7-422f-a15d-71b0abeed209","Type":"ContainerDied","Data":"bb65cd926393867221d9f8e0d3f44e41905476a926fd4e91f9e8c300eb09f0ba"} Oct 30 21:14:15.641615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.641592 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:15.802780 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:14:15.802747 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:14:15.844065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:15.844041 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:16.643945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:16.643910 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghxxm" event={"ID":"33b37892-bcc7-422f-a15d-71b0abeed209","Type":"ContainerStarted","Data":"cfe0228b472d797a1a3d5f76a194930d8f8494eebd7e8fec5cefb78620926f9b"} Oct 30 21:14:16.644349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:16.643990 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghxxm" event={"ID":"33b37892-bcc7-422f-a15d-71b0abeed209","Type":"ContainerStarted","Data":"0f4e1ebc0ae80ec6205945a2bf921188809c9f069ff016750e2d00f71f150140"} Oct 30 21:14:16.768148 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:16.768101 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ghxxm" podStartSLOduration=3.17412051 podStartE2EDuration="4.768087158s" totalImagesPullingTime="1.52490674s" totalInitContainerRuntime="69.059908ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.76765397 +0000 UTC m=+37.461136206" watchObservedRunningTime="2026-10-30 21:14:16.768087158 +0000 UTC m=+37.461569393" Oct 30 21:14:17.241273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:17.241235 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:17.438370 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:17.438340 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:17.627009 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:17.626977 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:17.902436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:17.902361 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.089690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:18.089659 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.574679 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:18.574654 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:14:19.254101 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:19.254070 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:19.456328 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:19.456300 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:19.582645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:19.582589 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:14:20.714635 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:20.714600 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.889953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:20.889925 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:14:20.916998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:20.916965 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.108610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:21.108538 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.353300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:21.353267 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.533985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:21.533938 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.719401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:21.719370 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.906091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:21.906066 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:22.243195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:22.243113 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:24.596749 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.596699 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" podUID="26902984-9b1a-478b-8613-6ae684818d37" containerName="registry" containerID="cri-o://735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019" gracePeriod=30 Oct 30 21:14:24.836205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.836177 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:14:24.873368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873338 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873380 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873412 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873444 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873474 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873491 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.873557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873528 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.874222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.873648 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") pod \"26902984-9b1a-478b-8613-6ae684818d37\" (UID: \"26902984-9b1a-478b-8613-6ae684818d37\") " Oct 30 21:14:24.874222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.874194 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:24.874346 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.874304 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:24.876194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.876163 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "kube-api-access-8jph5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:24.876317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.876236 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:24.876373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.876350 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:24.876410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.876373 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:24.876622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.876600 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:24.882568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.882538 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted" pod "26902984-9b1a-478b-8613-6ae684818d37" (UID: "26902984-9b1a-478b-8613-6ae684818d37"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:24.974896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974858 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jph5\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-kube-api-access-8jph5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.974896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974890 4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-bound-sa-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.974896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974900 4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26902984-9b1a-478b-8613-6ae684818d37-ca-trust-extracted\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.975138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974911 4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-image-registry-private-configuration\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.975138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974922 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-trusted-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.975138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974931 4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26902984-9b1a-478b-8613-6ae684818d37-registry-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.975138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974941 4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26902984-9b1a-478b-8613-6ae684818d37-installation-pull-secrets\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.975138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:24.974949 4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26902984-9b1a-478b-8613-6ae684818d37-registry-certificates\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:25.003890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.003847 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:25.193311 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.193222 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:25.688760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.688723 4918 generic.go:356] "Generic (PLEG): container finished" podID="26902984-9b1a-478b-8613-6ae684818d37" containerID="735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019" exitCode=0 Oct 30 21:14:25.689225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.688774 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" Oct 30 21:14:25.689225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.688811 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" event={"ID":"26902984-9b1a-478b-8613-6ae684818d37","Type":"ContainerDied","Data":"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019"} Oct 30 21:14:25.689225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.688872 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b77fdbb-tlfmz" event={"ID":"26902984-9b1a-478b-8613-6ae684818d37","Type":"ContainerDied","Data":"e6aa6a5361e05c1f3b7d189ceb0e44ee755b07e33eadf5349e1d7156be635eb1"} Oct 30 21:14:25.689225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.688905 4918 scope.go:122] "RemoveContainer" containerID="735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019" Oct 30 21:14:25.697433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.697414 4918 scope.go:122] "RemoveContainer" containerID="735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019" Oct 30 21:14:25.697740 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:14:25.697717 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019\": container with ID starting with 735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019 not found: ID does not exist" containerID="735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019" Oct 30 21:14:25.697824 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.697754 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019"} err="failed to get container status \"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019\": rpc error: code = NotFound desc = could not find container \"735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019\": container with ID starting with 735643b50ac4f425f83c52257183eb151b32902de78b413d521bb9016bb45019 not found: ID does not exist" Oct 30 21:14:25.889284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.889254 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:14:25.935698 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:25.935660 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d7b77fdbb-tlfmz"] Oct 30 21:14:26.943978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:26.943947 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:27.129439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:27.129407 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:27.425089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:27.425057 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26902984-9b1a-478b-8613-6ae684818d37" path="/var/lib/kubelet/pods/26902984-9b1a-478b-8613-6ae684818d37/volumes" Oct 30 21:14:28.245439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:28.245408 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.432400 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:28.432371 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:29.554730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:29.554689 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:29.765029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:29.764996 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:30.043483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:30.043453 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:33.801391 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:33.801361 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:33.987015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:33.986987 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:39.205295 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:39.205245 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:39.396462 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:39.396434 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:39.947464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:39.947429 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:40.145846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:40.145815 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:44.413805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:44.413767 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:44.610503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:44.610470 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:45.765595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:45.765564 4918 generic.go:356] "Generic (PLEG): container finished" podID="df904ba1-72fc-41f3-82d7-959374abe111" containerID="7566aa62e5f73fcdee0a2d9e618d9dbab649275949ec6fdf712efaf45b4e5b4a" exitCode=0 Oct 30 21:14:45.765976 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:45.765636 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" event={"ID":"df904ba1-72fc-41f3-82d7-959374abe111","Type":"ContainerDied","Data":"7566aa62e5f73fcdee0a2d9e618d9dbab649275949ec6fdf712efaf45b4e5b4a"} Oct 30 21:14:45.766015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:45.765993 4918 scope.go:122] "RemoveContainer" containerID="7566aa62e5f73fcdee0a2d9e618d9dbab649275949ec6fdf712efaf45b4e5b4a" Oct 30 21:14:45.801687 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:14:45.801665 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:14:45.909937 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:45.909888 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" podUID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" containerName="registry" containerID="cri-o://c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b" gracePeriod=30 Oct 30 21:14:46.156053 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.156027 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:14:46.221768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221690 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.221891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221772 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.221891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221805 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.221891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221836 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.222068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221922 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.222068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.221971 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.222068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.222006 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.222221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.222074 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") pod \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\" (UID: \"306c0704-8d54-4fb6-a6f9-1f25bdc32289\") " Oct 30 21:14:46.222286 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.222237 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:46.222757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.222650 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:46.224540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.224502 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:46.224631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.224547 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:46.224631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.224553 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:46.224631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.224613 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:46.224785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.224631 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "kube-api-access-6r2wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:46.228966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.228941 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted" pod "306c0704-8d54-4fb6-a6f9-1f25bdc32289" (UID: "306c0704-8d54-4fb6-a6f9-1f25bdc32289"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:46.322624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322595 4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-installation-pull-secrets\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322617 4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306c0704-8d54-4fb6-a6f9-1f25bdc32289-image-registry-private-configuration\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322627 4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322637 4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-registry-certificates\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322646 4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-bound-sa-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322654 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r2wd\" (UniqueName: \"kubernetes.io/projected/306c0704-8d54-4fb6-a6f9-1f25bdc32289-kube-api-access-6r2wd\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322667 4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306c0704-8d54-4fb6-a6f9-1f25bdc32289-ca-trust-extracted\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.322850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.322678 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306c0704-8d54-4fb6-a6f9-1f25bdc32289-trusted-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.770585 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.770549 4918 generic.go:356] "Generic (PLEG): container finished" podID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" containerID="c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b" exitCode=0 Oct 30 21:14:46.770998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.770599 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" Oct 30 21:14:46.770998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.770633 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" event={"ID":"306c0704-8d54-4fb6-a6f9-1f25bdc32289","Type":"ContainerDied","Data":"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b"} Oct 30 21:14:46.770998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.770683 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79dd8f4579-9l4tv" event={"ID":"306c0704-8d54-4fb6-a6f9-1f25bdc32289","Type":"ContainerDied","Data":"4d16429807999ff39fdddde0cfadad1da13a4a7be1f88055615b88ed57c85d45"} Oct 30 21:14:46.770998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.770718 4918 scope.go:122] "RemoveContainer" containerID="c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b" Oct 30 21:14:46.772590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.772565 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kw9fp" event={"ID":"df904ba1-72fc-41f3-82d7-959374abe111","Type":"ContainerStarted","Data":"e282a89c8a28403310dd55201cc36af26f86446ab6072c24424d81ef3ea6af25"} Oct 30 21:14:46.779838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.779816 4918 scope.go:122] "RemoveContainer" containerID="c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b" Oct 30 21:14:46.780116 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:14:46.780098 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b\": container with ID starting with c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b not found: ID does not exist" containerID="c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b" Oct 30 21:14:46.780163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:46.780125 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b"} err="failed to get container status \"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b\": rpc error: code = NotFound desc = could not find container \"c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b\": container with ID starting with c457016bbf64b84c49d4c03f0a3999885e49f092e598c4095b63d972ccb45d0b not found: ID does not exist" Oct 30 21:14:47.086356 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:47.086280 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:14:47.132738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:47.132691 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79dd8f4579-9l4tv"] Oct 30 21:14:47.425397 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:47.425364 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" path="/var/lib/kubelet/pods/306c0704-8d54-4fb6-a6f9-1f25bdc32289/volumes" Oct 30 21:14:49.790479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:49.790446 4918 generic.go:356] "Generic (PLEG): container finished" podID="e95e0daf-f36a-4951-8ead-9cf4ef55a928" containerID="d7b13dfe53cdc2d30339b681bdcf43b10217cf0ba5c13f12102ad410bc5d0443" exitCode=0 Oct 30 21:14:49.790845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:49.790508 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2q86m" event={"ID":"e95e0daf-f36a-4951-8ead-9cf4ef55a928","Type":"ContainerDied","Data":"d7b13dfe53cdc2d30339b681bdcf43b10217cf0ba5c13f12102ad410bc5d0443"} Oct 30 21:14:49.790845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:49.790831 4918 scope.go:122] "RemoveContainer" containerID="d7b13dfe53cdc2d30339b681bdcf43b10217cf0ba5c13f12102ad410bc5d0443" Oct 30 21:14:50.796144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:50.796111 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2q86m" event={"ID":"e95e0daf-f36a-4951-8ead-9cf4ef55a928","Type":"ContainerStarted","Data":"ce4dd359eddfffab2e59c6597ff7da01bb156602e62ded9f534db2d55aaadb95"} Oct 30 21:14:54.952165 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:54.952125 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:55.144029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:55.143990 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:58.209978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:58.209946 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:58.398834 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:14:58.398803 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:05.852305 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:05.852260 4918 generic.go:356] "Generic (PLEG): container finished" podID="ec41e273-2c37-467e-b55d-cef56c254d57" containerID="4eb41b005ec5b46fe67cfd723beaad7d37a804282f3992864b3b742c832a562e" exitCode=0 Oct 30 21:15:05.852686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:05.852331 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" event={"ID":"ec41e273-2c37-467e-b55d-cef56c254d57","Type":"ContainerDied","Data":"4eb41b005ec5b46fe67cfd723beaad7d37a804282f3992864b3b742c832a562e"} Oct 30 21:15:05.852686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:05.852636 4918 scope.go:122] "RemoveContainer" containerID="4eb41b005ec5b46fe67cfd723beaad7d37a804282f3992864b3b742c832a562e" Oct 30 21:15:06.858547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:06.858506 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rqm9s" event={"ID":"ec41e273-2c37-467e-b55d-cef56c254d57","Type":"ContainerStarted","Data":"b4cbc31cb0ef471c486e3a546282f1ea15842af9874adb0d9f759676e819212c"} Oct 30 21:15:15.799546 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:15:15.799513 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:15:26.827917 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:26.827871 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:27.316641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:27.316587 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:29.849247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:29.849211 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.171161 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:37.171127 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.576561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:37.576490 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.761775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:37.761743 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:45.799410 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:15:45.799372 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:15:49.975362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:49.975328 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:52.133903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:15:52.133858 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:15.799893 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:16:15.799854 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:16:30.211201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:30.211156 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:30.511752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:30.511664 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.491898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:44.491863 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:45.799195 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:16:45.799157 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:16:54.707453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:54.707418 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:54.898411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:54.898381 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:55.113192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:55.113151 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:55.296480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:55.296455 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:57.162100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:57.162065 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:57.345065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:16:57.345040 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:09.954778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:17:09.954746 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:10.152768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:17:10.152730 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:15.800815 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:17:15.800774 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:17:30.541230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:17:30.541158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:45.800226 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:17:45.800189 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:17:56.825663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:17:56.825626 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.799806 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:18:15.799762 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:18:30.877003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:30.876972 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:39.428012 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:39.427979 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:18:39.430156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:39.430128 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:18:39.433900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:39.433875 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:18:39.435802 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:39.435780 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:18:42.263519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:42.263484 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:42.478393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:42.478313 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:43.153877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.153835 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth"] Oct 30 21:18:43.154250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154225 4918 cpu_manager.go:414] "containerMap: removing container" podUID="26902984-9b1a-478b-8613-6ae684818d37" containerName="registry" Oct 30 21:18:43.154250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154246 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26902984-9b1a-478b-8613-6ae684818d37" containerName="registry" Oct 30 21:18:43.154442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154301 4918 cpu_manager.go:414] "containerMap: removing container" podUID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" containerName="registry" Oct 30 21:18:43.154442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154308 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" containerName="registry" Oct 30 21:18:43.154442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154411 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="306c0704-8d54-4fb6-a6f9-1f25bdc32289" containerName="registry" Oct 30 21:18:43.154442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.154427 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="26902984-9b1a-478b-8613-6ae684818d37" containerName="registry" Oct 30 21:18:43.157629 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.157606 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.188935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.188909 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lmq\" (UniqueName: \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.189051 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.188946 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.189051 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.188976 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.206928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.206904 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:18:43.206928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.206906 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:18:43.253994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.253965 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zjcsf\"" Oct 30 21:18:43.255004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.254987 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth"] Oct 30 21:18:43.255717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.255686 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp"] Oct 30 21:18:43.258968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.258951 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.289461 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289439 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lmq\" (UniqueName: \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.289864 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289483 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.289864 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289529 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.289864 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289557 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp8s\" (UniqueName: \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.289864 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289632 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.289864 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289700 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.290026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289932 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.290026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.289967 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.308628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.308608 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6xs96\"" Oct 30 21:18:43.320369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.320343 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp"] Oct 30 21:18:43.370232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.370211 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lmq\" (UniqueName: \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.390352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.390333 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.390455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.390365 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp8s\" (UniqueName: \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.390455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.390392 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.390770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.390753 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.390845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.390779 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.449613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.449556 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp8s\" (UniqueName: \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.473964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.473932 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:18:43.573906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.573881 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:18:43.834479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.834405 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:43.840733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.840689 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth"] Oct 30 21:18:43.844798 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:43.844771 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742c6569_8d01_45c5_8096_2ebaa985522f.slice/crio-4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c WatchSource:0}: Error finding container 4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c: Status 404 returned error can't find the container with id 4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c Oct 30 21:18:43.846820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.846803 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:18:43.937146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:43.937110 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp"] Oct 30 21:18:43.939371 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:43.939349 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07be4f1_390e_4a00_b99a_85befd69e30b.slice/crio-a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa WatchSource:0}: Error finding container a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa: Status 404 returned error can't find the container with id a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa Oct 30 21:18:44.023914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:44.023887 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:44.627235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:44.627199 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerStarted","Data":"a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa"} Oct 30 21:18:44.628298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:44.628273 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerStarted","Data":"4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c"} Oct 30 21:18:45.223279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.223245 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:45.449309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.449274 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr"] Oct 30 21:18:45.452944 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.452917 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.508734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.508652 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.508734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.508698 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.508949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.508915 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhdv\" (UniqueName: \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.549928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.549902 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-884wt\"" Oct 30 21:18:45.551068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.551043 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr"] Oct 30 21:18:45.609314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.609288 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhdv\" (UniqueName: \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.609456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.609351 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.609540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.609519 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.609754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.609730 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.609963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.609936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.668640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.668603 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhdv\" (UniqueName: \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.770815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:45.770484 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:18:45.812137 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:18:45.812098 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:18:46.150343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:46.150296 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr"] Oct 30 21:18:46.155861 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:46.155831 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac5100a_6362_4372_9b6a_b039dea7bb09.slice/crio-af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768 WatchSource:0}: Error finding container af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768: Status 404 returned error can't find the container with id af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768 Oct 30 21:18:46.639821 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:46.639785 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerStarted","Data":"af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768"} Oct 30 21:18:47.919337 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:47.919301 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr"] Oct 30 21:18:47.923438 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:47.923410 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.020026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.019997 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-6txp7\"" Oct 30 21:18:48.021875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.021846 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr"] Oct 30 21:18:48.024120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.024094 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw"] Oct 30 21:18:48.028131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.028108 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.033787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.033543 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.033787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.033603 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkwp\" (UniqueName: \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.033787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.033661 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.077818 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.077792 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-tkh9h\"" Oct 30 21:18:48.087432 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.087406 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw"] Oct 30 21:18:48.134078 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134035 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.134227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134131 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.134227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134202 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.134340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134264 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkwp\" (UniqueName: \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.134340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134303 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.134437 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134341 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcztc\" (UniqueName: \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.134791 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134769 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.135000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.134981 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.154250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.154225 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j"] Oct 30 21:18:48.157944 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.157924 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.194060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.193997 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkwp\" (UniqueName: \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.210221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.210198 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xqqxt\"" Oct 30 21:18:48.215004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.214916 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j"] Oct 30 21:18:48.222019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.221995 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh"] Oct 30 21:18:48.226503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.226113 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.235532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235506 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.235619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235553 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcztc\" (UniqueName: \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.235619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235588 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.235742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235633 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.235742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235658 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6hh\" (UniqueName: \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.235742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.235717 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.236155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.236112 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.236589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.236560 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.244055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.244013 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:18:48.282767 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.282718 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh"] Oct 30 21:18:48.299050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.299027 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcztc\" (UniqueName: \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.312093 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.312056 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-rprhg\"" Oct 30 21:18:48.336300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.336272 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.336419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.336323 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6hh\" (UniqueName: \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.336419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.336368 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88wq\" (UniqueName: \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.336419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.336409 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.336521 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.336439 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.337187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.337162 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.337293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.337273 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.337615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.337592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.347300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.347279 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:18:48.395241 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.395212 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6hh\" (UniqueName: \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.438024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.437989 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88wq\" (UniqueName: \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.438131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.438043 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.438131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.438086 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.438555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.438533 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.439094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.439063 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.477355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.477284 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:18:48.499964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.499894 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88wq\" (UniqueName: \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.549172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.549024 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:18:48.626289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.625786 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr"] Oct 30 21:18:48.652993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.652965 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerStarted","Data":"2d8ce58b35c35d11a8d5fb6cb8196ae89d4f6d95ab5d54bf01148b6c15873722"} Oct 30 21:18:48.726985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.726951 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw"] Oct 30 21:18:48.731224 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:48.731195 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956c8ce3_c567_456d_a73a_2c2971ec4495.slice/crio-925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7 WatchSource:0}: Error finding container 925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7: Status 404 returned error can't find the container with id 925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7 Oct 30 21:18:48.868166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.868135 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j"] Oct 30 21:18:48.871936 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:48.871900 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced32944_9c90_4502_8965_6a480dbf30b3.slice/crio-48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1 WatchSource:0}: Error finding container 48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1: Status 404 returned error can't find the container with id 48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1 Oct 30 21:18:48.939322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:48.939281 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh"] Oct 30 21:18:48.943152 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:48.943122 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode733744d_e7db_4b07_8a60_4dc10dc14b45.slice/crio-500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9 WatchSource:0}: Error finding container 500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9: Status 404 returned error can't find the container with id 500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9 Oct 30 21:18:49.047126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.047049 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x"] Oct 30 21:18:49.053622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.052988 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.111168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.111097 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x"] Oct 30 21:18:49.144427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.144393 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkggb\" (UniqueName: \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.144586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.144452 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.144586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.144484 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.246059 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.246021 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkggb\" (UniqueName: \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.246202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.246083 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.246202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.246110 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.246536 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.246504 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.247067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.247042 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.278963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.278934 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7"] Oct 30 21:18:49.283137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.283118 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.310482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.310426 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkggb\" (UniqueName: \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.340542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.339628 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7"] Oct 30 21:18:49.347468 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.347435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.347607 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.347540 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.347607 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.347578 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ff82\" (UniqueName: \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.373354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.373326 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:18:49.448115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.448083 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.448286 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.448206 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.448373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.448285 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ff82\" (UniqueName: \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.449015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.448990 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.449421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.449400 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.507405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.507371 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ff82\" (UniqueName: \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.601399 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.601330 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:18:49.662223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.662146 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerStarted","Data":"48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1"} Oct 30 21:18:49.666092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.666058 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" event={"ID":"e733744d-e7db-4b07-8a60-4dc10dc14b45","Type":"ContainerStarted","Data":"500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9"} Oct 30 21:18:49.668586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.668560 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerStarted","Data":"925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7"} Oct 30 21:18:49.767530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.767499 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x"] Oct 30 21:18:49.774777 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:49.774745 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd514dcd8_4b76_40ac_b323_836fd1e7b15d.slice/crio-165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9 WatchSource:0}: Error finding container 165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9: Status 404 returned error can't find the container with id 165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9 Oct 30 21:18:49.969856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.969796 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:49.994540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:49.994511 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7"] Oct 30 21:18:50.222969 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:50.222899 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:50.688129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:50.688057 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerStarted","Data":"9c90169ea5da1395c717023e01742afb3576baa6130047a200e86643668295ae"} Oct 30 21:18:50.693445 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:50.693379 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" event={"ID":"d514dcd8-4b76-40ac-b323-836fd1e7b15d","Type":"ContainerStarted","Data":"165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9"} Oct 30 21:18:51.448196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.448165 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx"] Oct 30 21:18:51.452781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.452304 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.502718 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.502653 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z4vl5\"" Oct 30 21:18:51.519829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.519780 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:18:51.552293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.552215 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx"] Oct 30 21:18:51.571724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571677 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef085da-a319-4c09-a236-af0467dc72ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.571876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571801 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.571876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571830 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.571876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571856 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.572028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef085da-a319-4c09-a236-af0467dc72ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.572028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571933 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.572028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571960 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7nf\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-kube-api-access-ds7nf\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.572028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.571998 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.572028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.572025 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.582923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.582838 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673003 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673085 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673118 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef085da-a319-4c09-a236-af0467dc72ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673156 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673187 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7nf\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-kube-api-access-ds7nf\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673227 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673254 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.673296 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef085da-a319-4c09-a236-af0467dc72ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.674104 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef085da-a319-4c09-a236-af0467dc72ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.674376 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.675485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.674906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.677496 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.676448 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.677496 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.676930 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.678495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.678393 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef085da-a319-4c09-a236-af0467dc72ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.685699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.685630 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef085da-a319-4c09-a236-af0467dc72ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.745689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.745616 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6"] Oct 30 21:18:51.748688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.748648 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.750660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.750639 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.754227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.754182 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7nf\" (UniqueName: \"kubernetes.io/projected/aef085da-a319-4c09-a236-af0467dc72ec-kube-api-access-ds7nf\") pod \"maas-default-gateway-openshift-default-67bf986f99-smwqx\" (UID: \"aef085da-a319-4c09-a236-af0467dc72ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.771541 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.771519 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:18:51.774816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.774793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.774927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.774894 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.774991 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.774936 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775015 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5hj\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-kube-api-access-kx5hj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775057 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775084 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775120 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775169 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.775310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.775204 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.800867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.800831 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-nk6d7\"" Oct 30 21:18:51.808996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.808973 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6"] Oct 30 21:18:51.812505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.812462 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876134 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876185 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876225 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876279 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5hj\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-kube-api-access-kx5hj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876317 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876344 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876376 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876423 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.877155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.876457 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.879135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.878236 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.880368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.879333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.880368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.879590 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.880368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.880309 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.881003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.880931 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.886376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.884662 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.926498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.926440 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.947006 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.946936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:51.948235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:51.948108 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5hj\" (UniqueName: \"kubernetes.io/projected/f2aaa457-d02c-4c2e-9db0-c5769374a3b2-kube-api-access-kx5hj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6\" (UID: \"f2aaa457-d02c-4c2e-9db0-c5769374a3b2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:52.053020 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.052814 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:52.087882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.087514 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:18:52.251990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.251485 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx"] Oct 30 21:18:52.256848 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:18:52.256817 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef085da_a319_4c09_a236_af0467dc72ec.slice/crio-9ce2985eed7e5d67fe29490acb0961adb27697d4b6c2f1986b10bdcaf0574a39 WatchSource:0}: Error finding container 9ce2985eed7e5d67fe29490acb0961adb27697d4b6c2f1986b10bdcaf0574a39: Status 404 returned error can't find the container with id 9ce2985eed7e5d67fe29490acb0961adb27697d4b6c2f1986b10bdcaf0574a39 Oct 30 21:18:52.514017 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.513908 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6"] Oct 30 21:18:52.713881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.713742 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" event={"ID":"aef085da-a319-4c09-a236-af0467dc72ec","Type":"ContainerStarted","Data":"9ce2985eed7e5d67fe29490acb0961adb27697d4b6c2f1986b10bdcaf0574a39"} Oct 30 21:18:52.719237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:52.719201 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" event={"ID":"f2aaa457-d02c-4c2e-9db0-c5769374a3b2","Type":"ContainerStarted","Data":"c16caa32ca27077a3c6b8f7cbb105695d25d0ae92fb79decd7be80b6b9b2bbba"} Oct 30 21:18:55.594763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:55.594638 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.045106 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:56.044988 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.241813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:56.241722 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.783652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:56.783334 4918 generic.go:356] "Generic (PLEG): container finished" podID="742c6569-8d01-45c5-8096-2ebaa985522f" containerID="bb6a2ecc9afd717baf147dc5d4eade34d9fcc9186a45707b3e05cc7aed4d7b65" exitCode=0 Oct 30 21:18:56.783652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:56.783384 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerDied","Data":"bb6a2ecc9afd717baf147dc5d4eade34d9fcc9186a45707b3e05cc7aed4d7b65"} Oct 30 21:18:58.816569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:58.815528 4918 generic.go:356] "Generic (PLEG): container finished" podID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerID="9552514e8359816766a1cea1f91a408b8b1faf654cafa24368fbeb9efd50fd10" exitCode=0 Oct 30 21:18:58.816569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:18:58.815634 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerDied","Data":"9552514e8359816766a1cea1f91a408b8b1faf654cafa24368fbeb9efd50fd10"} Oct 30 21:19:00.837527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:00.837496 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerID="23d50494ce5ba6f52cb7213ebbae5047e11888a073993c3e17d0f51355396ac6" exitCode=0 Oct 30 21:19:00.837931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:00.837583 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerDied","Data":"23d50494ce5ba6f52cb7213ebbae5047e11888a073993c3e17d0f51355396ac6"} Oct 30 21:19:00.843040 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:00.842618 4918 generic.go:356] "Generic (PLEG): container finished" podID="ced32944-9c90-4502-8965-6a480dbf30b3" containerID="0ea8d31798952ac97b1901fd7aa97b9d337af6d9f23e4ff75d48c5d3f23b9170" exitCode=0 Oct 30 21:19:00.843040 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:00.842695 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerDied","Data":"0ea8d31798952ac97b1901fd7aa97b9d337af6d9f23e4ff75d48c5d3f23b9170"} Oct 30 21:19:01.847746 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.847716 4918 generic.go:356] "Generic (PLEG): container finished" podID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerID="8f60d7164d5510a431bb1f0e3cab40996896d156d480c876373abc0d681c006e" exitCode=0 Oct 30 21:19:01.848079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.847805 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" event={"ID":"e733744d-e7db-4b07-8a60-4dc10dc14b45","Type":"ContainerDied","Data":"8f60d7164d5510a431bb1f0e3cab40996896d156d480c876373abc0d681c006e"} Oct 30 21:19:01.849323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.849303 4918 generic.go:356] "Generic (PLEG): container finished" podID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerID="4cc6830d304fe53fbcaee2750178352c10a6ee5fda0d061aa44357dbf92ef92b" exitCode=0 Oct 30 21:19:01.849395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.849377 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerDied","Data":"4cc6830d304fe53fbcaee2750178352c10a6ee5fda0d061aa44357dbf92ef92b"} Oct 30 21:19:01.850769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.850747 4918 generic.go:356] "Generic (PLEG): container finished" podID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerID="601af9a4abebe643731bce095d8748aa40efcd39e333d63ad8acfdb87b3ad665" exitCode=0 Oct 30 21:19:01.850869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:01.850827 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" event={"ID":"d514dcd8-4b76-40ac-b323-836fd1e7b15d","Type":"ContainerDied","Data":"601af9a4abebe643731bce095d8748aa40efcd39e333d63ad8acfdb87b3ad665"} Oct 30 21:19:02.449466 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.449437 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:02.449560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.449499 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:02.449560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.449528 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:02.857362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.857285 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerID="27c8250504bedd052904b4086160044501b8f368c5df8be4e0ca30a4c8cb0d28" exitCode=0 Oct 30 21:19:02.857879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.857379 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerDied","Data":"27c8250504bedd052904b4086160044501b8f368c5df8be4e0ca30a4c8cb0d28"} Oct 30 21:19:02.859383 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.859310 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" event={"ID":"aef085da-a319-4c09-a236-af0467dc72ec","Type":"ContainerStarted","Data":"e6c0a7f8f2bac818adcbb861313ee3eaf43070ab383b41e85fad8119f6695023"} Oct 30 21:19:02.861229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.861202 4918 generic.go:356] "Generic (PLEG): container finished" podID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerID="3b383ee4413d4c5a990ecfcc8a114e1ce8b7e374997c81ee3b4ae6365d0fa5b1" exitCode=0 Oct 30 21:19:02.861337 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.861283 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerDied","Data":"3b383ee4413d4c5a990ecfcc8a114e1ce8b7e374997c81ee3b4ae6365d0fa5b1"} Oct 30 21:19:02.953147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.953104 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:02.953259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.953180 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:02.953259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:02.953218 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:19:03.199909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.199844 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podStartSLOduration=2.015431372 podStartE2EDuration="12.199824252s" totalImagesPullingTime="10.18439288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:03.197227547 +0000 UTC m=+323.890709784" watchObservedRunningTime="2026-10-30 21:19:03.199824252 +0000 UTC m=+323.893306489" Oct 30 21:19:03.772265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.772190 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:19:03.777743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.777716 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:19:03.868742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.868687 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" event={"ID":"f2aaa457-d02c-4c2e-9db0-c5769374a3b2","Type":"ContainerStarted","Data":"bb28409676a44826c27bbce42a2ec341da0806c85e173c89b4ba6783a6900c8b"} Oct 30 21:19:03.870860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.870833 4918 generic.go:356] "Generic (PLEG): container finished" podID="ced32944-9c90-4502-8965-6a480dbf30b3" containerID="aa5c0909da1f5f18d341423e1afdc1fb406cf30aeacc54e42e3e28ff76958773" exitCode=0 Oct 30 21:19:03.870992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.870910 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerDied","Data":"aa5c0909da1f5f18d341423e1afdc1fb406cf30aeacc54e42e3e28ff76958773"} Oct 30 21:19:03.871119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.871074 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:19:03.872110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:03.872092 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:19:04.010340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.010285 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" podStartSLOduration=2.587632584 podStartE2EDuration="13.010267935s" totalImagesPullingTime="10.422635351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:04.009229184 +0000 UTC m=+324.702711414" watchObservedRunningTime="2026-10-30 21:19:04.010267935 +0000 UTC m=+324.703750174" Oct 30 21:19:04.087989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.087962 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:19:04.092448 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.092425 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:19:04.878623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.878583 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerID="5c3a547e24a6f5e1fa9d5962a3f3bc9fb37890e8211f7339b78adf7f5d445230" exitCode=0 Oct 30 21:19:04.879110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.878664 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerDied","Data":"5c3a547e24a6f5e1fa9d5962a3f3bc9fb37890e8211f7339b78adf7f5d445230"} Oct 30 21:19:04.881237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.881206 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerID="5af5a03d9f6fe8b87848e999635b6d0e184fca544fbd54b78de41cf5a2434aed" exitCode=0 Oct 30 21:19:04.881328 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.881271 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerDied","Data":"5af5a03d9f6fe8b87848e999635b6d0e184fca544fbd54b78de41cf5a2434aed"} Oct 30 21:19:04.883202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.883183 4918 generic.go:356] "Generic (PLEG): container finished" podID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerID="22b2ac76ab065705263bfe317e8e55a4a0295ed28dee95778a1e07802615ede0" exitCode=0 Oct 30 21:19:04.883279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.883212 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" event={"ID":"e733744d-e7db-4b07-8a60-4dc10dc14b45","Type":"ContainerDied","Data":"22b2ac76ab065705263bfe317e8e55a4a0295ed28dee95778a1e07802615ede0"} Oct 30 21:19:04.885574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.885553 4918 generic.go:356] "Generic (PLEG): container finished" podID="742c6569-8d01-45c5-8096-2ebaa985522f" containerID="f88d1dea202dcb383c9b42def4b03852ca5276c199ddd71194cb3a580ef454b8" exitCode=0 Oct 30 21:19:04.885691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.885585 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerDied","Data":"f88d1dea202dcb383c9b42def4b03852ca5276c199ddd71194cb3a580ef454b8"} Oct 30 21:19:04.887656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.887636 4918 generic.go:356] "Generic (PLEG): container finished" podID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerID="bf9aab50ad21814607fbd16f11468471ccdf22298b64c6f79e7368ce9758ef1d" exitCode=0 Oct 30 21:19:04.887816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.887757 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerDied","Data":"bf9aab50ad21814607fbd16f11468471ccdf22298b64c6f79e7368ce9758ef1d"} Oct 30 21:19:04.889668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.889646 4918 generic.go:356] "Generic (PLEG): container finished" podID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerID="da7415215fc0bd27bc1eb9b179803ded785ac75da4c1d4b3df076c86d79078a5" exitCode=0 Oct 30 21:19:04.889782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.889730 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" event={"ID":"d514dcd8-4b76-40ac-b323-836fd1e7b15d","Type":"ContainerDied","Data":"da7415215fc0bd27bc1eb9b179803ded785ac75da4c1d4b3df076c86d79078a5"} Oct 30 21:19:04.890147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.890113 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:19:04.890983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:04.890965 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6" Oct 30 21:19:05.900728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:05.900313 4918 generic.go:356] "Generic (PLEG): container finished" podID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerID="4e7feeb5d9ccc5d811911e74d55c2484d2469ab47c7c9e07a74390b5a603e79d" exitCode=0 Oct 30 21:19:05.900728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:05.900409 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerDied","Data":"4e7feeb5d9ccc5d811911e74d55c2484d2469ab47c7c9e07a74390b5a603e79d"} Oct 30 21:19:05.906287 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:05.905487 4918 generic.go:356] "Generic (PLEG): container finished" podID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerID="d9a69f0702e667e8d9295d9e642d616403935b9cff27ced25b19d1ae4afd3f90" exitCode=0 Oct 30 21:19:05.906287 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:05.905788 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerDied","Data":"d9a69f0702e667e8d9295d9e642d616403935b9cff27ced25b19d1ae4afd3f90"} Oct 30 21:19:15.827659 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:19:15.827504 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:19:22.085668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:22.085630 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerStarted","Data":"14f2144e58d9f3707dd6ad1c4e700ca836a00a8cf4f00c1eb6a59602c667afbf"} Oct 30 21:19:22.200502 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:22.200437 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" podStartSLOduration=2.64228911 podStartE2EDuration="34.200417059s" totalImagesPullingTime="31.558127949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:22.199448939 +0000 UTC m=+342.892931175" watchObservedRunningTime="2026-10-30 21:19:22.200417059 +0000 UTC m=+342.893899296" Oct 30 21:19:23.105650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:23.105128 4918 generic.go:356] "Generic (PLEG): container finished" podID="ced32944-9c90-4502-8965-6a480dbf30b3" containerID="14f2144e58d9f3707dd6ad1c4e700ca836a00a8cf4f00c1eb6a59602c667afbf" exitCode=0 Oct 30 21:19:23.105650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:23.105244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerDied","Data":"14f2144e58d9f3707dd6ad1c4e700ca836a00a8cf4f00c1eb6a59602c667afbf"} Oct 30 21:19:24.295041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.295010 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:19:24.421778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.421750 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\" (UniqueName: \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\") pod \"ced32944-9c90-4502-8965-6a480dbf30b3\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " Oct 30 21:19:24.421910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.421848 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\") pod \"ced32944-9c90-4502-8965-6a480dbf30b3\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " Oct 30 21:19:24.421910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.421897 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\") pod \"ced32944-9c90-4502-8965-6a480dbf30b3\" (UID: \"ced32944-9c90-4502-8965-6a480dbf30b3\") " Oct 30 21:19:24.423071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.423044 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle" pod "ced32944-9c90-4502-8965-6a480dbf30b3" (UID: "ced32944-9c90-4502-8965-6a480dbf30b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:24.431107 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.431079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util" pod "ced32944-9c90-4502-8965-6a480dbf30b3" (UID: "ced32944-9c90-4502-8965-6a480dbf30b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:24.435484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.435458 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh" pod "ced32944-9c90-4502-8965-6a480dbf30b3" (UID: "ced32944-9c90-4502-8965-6a480dbf30b3"). InnerVolumeSpecName "kube-api-access-dv6hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:24.522760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.522730 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.522760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.522762 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced32944-9c90-4502-8965-6a480dbf30b3-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.522936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:24.522778 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6hh\" (UniqueName: \"kubernetes.io/projected/ced32944-9c90-4502-8965-6a480dbf30b3-kube-api-access-dv6hh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:25.118303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.118272 4918 generic.go:356] "Generic (PLEG): container finished" podID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerID="a80888681e40bf819ed4f070bc00ce3aa2868da262c615a00011454fae8aa4c8" exitCode=0 Oct 30 21:19:25.118456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.118352 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" event={"ID":"e733744d-e7db-4b07-8a60-4dc10dc14b45","Type":"ContainerDied","Data":"a80888681e40bf819ed4f070bc00ce3aa2868da262c615a00011454fae8aa4c8"} Oct 30 21:19:25.120612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.120590 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerStarted","Data":"5603f7bcd1da90cd5523a984e0f3176ed004cfd9e538fd27a5292aa1b0f763b5"} Oct 30 21:19:25.122890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.122869 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerStarted","Data":"2dd18eb5c34f145672c3df428b792317b4d3829dddfd0f3c7d0eb279e3614cc9"} Oct 30 21:19:25.124970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.124950 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerStarted","Data":"374c068f687fa84bf532cd6ca9c822e46a006186622c548b8215793f275e61b7"} Oct 30 21:19:25.126654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.126636 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" event={"ID":"ced32944-9c90-4502-8965-6a480dbf30b3","Type":"ContainerDied","Data":"48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1"} Oct 30 21:19:25.126783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.126665 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a6849142736baeabf03778a64a9af662cbd70fef4fac4a5dff6ee6134ff5d1" Oct 30 21:19:25.126783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.126687 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwsm9j" Oct 30 21:19:25.346716 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.346659 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" podStartSLOduration=3.267875387 podStartE2EDuration="38.346645858s" totalImagesPullingTime="35.078770471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:25.346081965 +0000 UTC m=+346.039564204" watchObservedRunningTime="2026-10-30 21:19:25.346645858 +0000 UTC m=+346.040128093" Oct 30 21:19:25.510201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.510118 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" podStartSLOduration=3.359797928 podStartE2EDuration="40.510103829s" totalImagesPullingTime="37.150305901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:25.509442888 +0000 UTC m=+346.202925124" watchObservedRunningTime="2026-10-30 21:19:25.510103829 +0000 UTC m=+346.203586081" Oct 30 21:19:25.622875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:25.622781 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" podStartSLOduration=3.230332422 podStartE2EDuration="42.622760007s" totalImagesPullingTime="39.392427585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:25.622110132 +0000 UTC m=+346.315592372" watchObservedRunningTime="2026-10-30 21:19:25.622760007 +0000 UTC m=+346.316242244" Oct 30 21:19:26.133337 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.133307 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerStarted","Data":"a4ab1635caeac412c67b4b5a4d687d0a07f7bb88cb9ef4e15a4c5c12878167bc"} Oct 30 21:19:26.135224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.135197 4918 generic.go:356] "Generic (PLEG): container finished" podID="742c6569-8d01-45c5-8096-2ebaa985522f" containerID="5603f7bcd1da90cd5523a984e0f3176ed004cfd9e538fd27a5292aa1b0f763b5" exitCode=0 Oct 30 21:19:26.135354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.135273 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerDied","Data":"5603f7bcd1da90cd5523a984e0f3176ed004cfd9e538fd27a5292aa1b0f763b5"} Oct 30 21:19:26.137353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.137332 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerStarted","Data":"d01ca7e47a524dbd443ecca4083732304e15df2e30706b47479b7d3e0fc8bfa1"} Oct 30 21:19:26.139243 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.139224 4918 generic.go:356] "Generic (PLEG): container finished" podID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerID="86de9e235cbbee2ee9ac2d64f67a89892c34f7726766fb646650ad86f84e86d0" exitCode=0 Oct 30 21:19:26.139365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.139343 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" event={"ID":"d514dcd8-4b76-40ac-b323-836fd1e7b15d","Type":"ContainerDied","Data":"86de9e235cbbee2ee9ac2d64f67a89892c34f7726766fb646650ad86f84e86d0"} Oct 30 21:19:26.141265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.141248 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerID="2dd18eb5c34f145672c3df428b792317b4d3829dddfd0f3c7d0eb279e3614cc9" exitCode=0 Oct 30 21:19:26.141355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.141313 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerDied","Data":"2dd18eb5c34f145672c3df428b792317b4d3829dddfd0f3c7d0eb279e3614cc9"} Oct 30 21:19:26.143171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.143153 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerID="374c068f687fa84bf532cd6ca9c822e46a006186622c548b8215793f275e61b7" exitCode=0 Oct 30 21:19:26.143261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.143198 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerDied","Data":"374c068f687fa84bf532cd6ca9c822e46a006186622c548b8215793f275e61b7"} Oct 30 21:19:26.145195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.145176 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerStarted","Data":"cc2a47ecf7b43df52b388b5512b9bfd8df750d0637615ab778b6388bf115fc59"} Oct 30 21:19:26.247217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.247156 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" podStartSLOduration=2.4724839 podStartE2EDuration="37.247138778s" totalImagesPullingTime="34.774654878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:26.246931241 +0000 UTC m=+346.940413478" watchObservedRunningTime="2026-10-30 21:19:26.247138778 +0000 UTC m=+346.940621014" Oct 30 21:19:26.279139 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.279116 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:19:26.338076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.338050 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\") pod \"e733744d-e7db-4b07-8a60-4dc10dc14b45\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " Oct 30 21:19:26.338156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.338096 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\" (UniqueName: \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\") pod \"e733744d-e7db-4b07-8a60-4dc10dc14b45\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " Oct 30 21:19:26.338156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.338133 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\") pod \"e733744d-e7db-4b07-8a60-4dc10dc14b45\" (UID: \"e733744d-e7db-4b07-8a60-4dc10dc14b45\") " Oct 30 21:19:26.338596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.338574 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle" pod "e733744d-e7db-4b07-8a60-4dc10dc14b45" (UID: "e733744d-e7db-4b07-8a60-4dc10dc14b45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:26.340196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.340169 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq" pod "e733744d-e7db-4b07-8a60-4dc10dc14b45" (UID: "e733744d-e7db-4b07-8a60-4dc10dc14b45"). InnerVolumeSpecName "kube-api-access-j88wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:26.343015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.342997 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util" pod "e733744d-e7db-4b07-8a60-4dc10dc14b45" (UID: "e733744d-e7db-4b07-8a60-4dc10dc14b45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:26.438681 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.438660 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:26.438961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.438682 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88wq\" (UniqueName: \"kubernetes.io/projected/e733744d-e7db-4b07-8a60-4dc10dc14b45-kube-api-access-j88wq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:26.438961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.438694 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e733744d-e7db-4b07-8a60-4dc10dc14b45-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:26.697424 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.697346 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" podStartSLOduration=3.848350604 podStartE2EDuration="43.697334336s" totalImagesPullingTime="39.848983732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:26.696594842 +0000 UTC m=+347.390077079" watchObservedRunningTime="2026-10-30 21:19:26.697334336 +0000 UTC m=+347.390816571" Oct 30 21:19:26.921096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:26.921053 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" podStartSLOduration=4.510040226 podStartE2EDuration="39.921041313s" totalImagesPullingTime="35.411001087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:26.920675902 +0000 UTC m=+347.614158149" watchObservedRunningTime="2026-10-30 21:19:26.921041313 +0000 UTC m=+347.614523548" Oct 30 21:19:27.151541 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.151511 4918 generic.go:356] "Generic (PLEG): container finished" podID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerID="cc2a47ecf7b43df52b388b5512b9bfd8df750d0637615ab778b6388bf115fc59" exitCode=0 Oct 30 21:19:27.151693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.151589 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerDied","Data":"cc2a47ecf7b43df52b388b5512b9bfd8df750d0637615ab778b6388bf115fc59"} Oct 30 21:19:27.153329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.153311 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" Oct 30 21:19:27.153449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.153309 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lknjh" event={"ID":"e733744d-e7db-4b07-8a60-4dc10dc14b45","Type":"ContainerDied","Data":"500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9"} Oct 30 21:19:27.153449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.153420 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500d602321da0c4c8f8cb8017a1a9f7c2b6ff23cae09fa777255d3e6a0b34db9" Oct 30 21:19:27.154804 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.154788 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:19:27.155467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.155439 4918 generic.go:356] "Generic (PLEG): container finished" podID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerID="a4ab1635caeac412c67b4b5a4d687d0a07f7bb88cb9ef4e15a4c5c12878167bc" exitCode=1 Oct 30 21:19:27.155545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.155511 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerDied","Data":"a4ab1635caeac412c67b4b5a4d687d0a07f7bb88cb9ef4e15a4c5c12878167bc"} Oct 30 21:19:27.157546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.157524 4918 generic.go:356] "Generic (PLEG): container finished" podID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerID="d01ca7e47a524dbd443ecca4083732304e15df2e30706b47479b7d3e0fc8bfa1" exitCode=0 Oct 30 21:19:27.157627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.157605 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerDied","Data":"d01ca7e47a524dbd443ecca4083732304e15df2e30706b47479b7d3e0fc8bfa1"} Oct 30 21:19:27.313208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.313186 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:19:27.346380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.346356 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\" (UniqueName: \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\") pod \"742c6569-8d01-45c5-8096-2ebaa985522f\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " Oct 30 21:19:27.346475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.346395 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\") pod \"742c6569-8d01-45c5-8096-2ebaa985522f\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " Oct 30 21:19:27.346475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.346466 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\") pod \"742c6569-8d01-45c5-8096-2ebaa985522f\" (UID: \"742c6569-8d01-45c5-8096-2ebaa985522f\") " Oct 30 21:19:27.346965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.346937 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle" pod "742c6569-8d01-45c5-8096-2ebaa985522f" (UID: "742c6569-8d01-45c5-8096-2ebaa985522f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.348413 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.348386 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq" pod "742c6569-8d01-45c5-8096-2ebaa985522f" (UID: "742c6569-8d01-45c5-8096-2ebaa985522f"). InnerVolumeSpecName "kube-api-access-z8lmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:27.352212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.352191 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util" pod "742c6569-8d01-45c5-8096-2ebaa985522f" (UID: "742c6569-8d01-45c5-8096-2ebaa985522f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.355484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.355465 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:19:27.358483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.358464 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:19:27.361194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.361178 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:19:27.447070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447047 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\" (UniqueName: \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\") pod \"3ac5100a-6362-4372-9b6a-b039dea7bb09\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447102 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\") pod \"a7aecce0-a0ac-4e6d-a355-1207901414b6\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447122 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\" (UniqueName: \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\") pod \"a7aecce0-a0ac-4e6d-a355-1207901414b6\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447147 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\") pod \"3ac5100a-6362-4372-9b6a-b039dea7bb09\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447165 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\") pod \"3ac5100a-6362-4372-9b6a-b039dea7bb09\" (UID: \"3ac5100a-6362-4372-9b6a-b039dea7bb09\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447189 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\") pod \"a7aecce0-a0ac-4e6d-a355-1207901414b6\" (UID: \"a7aecce0-a0ac-4e6d-a355-1207901414b6\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447222 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\" (UniqueName: \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\") pod \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447244 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\") pod \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " Oct 30 21:19:27.447419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447283 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\") pod \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\" (UID: \"d514dcd8-4b76-40ac-b323-836fd1e7b15d\") " Oct 30 21:19:27.447841 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447481 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.447841 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447503 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lmq\" (UniqueName: \"kubernetes.io/projected/742c6569-8d01-45c5-8096-2ebaa985522f-kube-api-access-z8lmq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.447841 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447518 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742c6569-8d01-45c5-8096-2ebaa985522f-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.447981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447664 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle" pod "a7aecce0-a0ac-4e6d-a355-1207901414b6" (UID: "a7aecce0-a0ac-4e6d-a355-1207901414b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.447981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.447899 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle" pod "d514dcd8-4b76-40ac-b323-836fd1e7b15d" (UID: "d514dcd8-4b76-40ac-b323-836fd1e7b15d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.449625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.449604 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle" pod "3ac5100a-6362-4372-9b6a-b039dea7bb09" (UID: "3ac5100a-6362-4372-9b6a-b039dea7bb09"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.454536 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.454512 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util" pod "3ac5100a-6362-4372-9b6a-b039dea7bb09" (UID: "3ac5100a-6362-4372-9b6a-b039dea7bb09"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.455809 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.455781 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util" pod "a7aecce0-a0ac-4e6d-a355-1207901414b6" (UID: "a7aecce0-a0ac-4e6d-a355-1207901414b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.456174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.456158 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util" pod "d514dcd8-4b76-40ac-b323-836fd1e7b15d" (UID: "d514dcd8-4b76-40ac-b323-836fd1e7b15d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:27.462992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.462971 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb" pod "d514dcd8-4b76-40ac-b323-836fd1e7b15d" (UID: "d514dcd8-4b76-40ac-b323-836fd1e7b15d"). InnerVolumeSpecName "kube-api-access-dkggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:27.477049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.477023 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp" pod "a7aecce0-a0ac-4e6d-a355-1207901414b6" (UID: "a7aecce0-a0ac-4e6d-a355-1207901414b6"). InnerVolumeSpecName "kube-api-access-tbkwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:27.477049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.477035 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv" pod "3ac5100a-6362-4372-9b6a-b039dea7bb09" (UID: "3ac5100a-6362-4372-9b6a-b039dea7bb09"). InnerVolumeSpecName "kube-api-access-jfhdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:27.548739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548720 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548741 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkwp\" (UniqueName: \"kubernetes.io/projected/a7aecce0-a0ac-4e6d-a355-1207901414b6-kube-api-access-tbkwp\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548752 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548760 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ac5100a-6362-4372-9b6a-b039dea7bb09-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548767 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7aecce0-a0ac-4e6d-a355-1207901414b6-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548775 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkggb\" (UniqueName: \"kubernetes.io/projected/d514dcd8-4b76-40ac-b323-836fd1e7b15d-kube-api-access-dkggb\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548782 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548790 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d514dcd8-4b76-40ac-b323-836fd1e7b15d-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:27.548831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:27.548798 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhdv\" (UniqueName: \"kubernetes.io/projected/3ac5100a-6362-4372-9b6a-b039dea7bb09-kube-api-access-jfhdv\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.164060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.164020 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" event={"ID":"3ac5100a-6362-4372-9b6a-b039dea7bb09","Type":"ContainerDied","Data":"af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768"} Oct 30 21:19:28.164060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.164048 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tghnr" Oct 30 21:19:28.164240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.164084 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af34131cb678403b89f76b69741d2a46001f45b27d8c3554e8d5643c4cdc8768" Oct 30 21:19:28.165993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.165969 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" Oct 30 21:19:28.166143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.166009 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wrbtr" event={"ID":"a7aecce0-a0ac-4e6d-a355-1207901414b6","Type":"ContainerDied","Data":"2d8ce58b35c35d11a8d5fb6cb8196ae89d4f6d95ab5d54bf01148b6c15873722"} Oct 30 21:19:28.166143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.166046 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8ce58b35c35d11a8d5fb6cb8196ae89d4f6d95ab5d54bf01148b6c15873722" Oct 30 21:19:28.168088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.168066 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" Oct 30 21:19:28.168088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.168078 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cpjth" event={"ID":"742c6569-8d01-45c5-8096-2ebaa985522f","Type":"ContainerDied","Data":"4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c"} Oct 30 21:19:28.168271 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.168119 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eaae99cf95e8d6d75681e8f18b163f56a11b8666df05352d24f58d8b04dab3c" Oct 30 21:19:28.170574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.170553 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" Oct 30 21:19:28.172381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.172358 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hph9x" event={"ID":"d514dcd8-4b76-40ac-b323-836fd1e7b15d","Type":"ContainerDied","Data":"165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9"} Oct 30 21:19:28.172452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.172387 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165943299b40dfcebaccd5899c7b25fee8a58260e480e76cccb452a94c49aee9" Oct 30 21:19:28.313476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.313451 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:19:28.326240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.326221 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:19:28.328676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.328660 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:19:28.349547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.349529 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:19:28.356028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356007 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\") pod \"735d7add-e2e7-4f2a-be19-2585f3bc731c\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " Oct 30 21:19:28.356120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356051 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\" (UniqueName: \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\") pod \"735d7add-e2e7-4f2a-be19-2585f3bc731c\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " Oct 30 21:19:28.356120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356072 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\") pod \"956c8ce3-c567-456d-a73a-2c2971ec4495\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " Oct 30 21:19:28.356233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356121 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\") pod \"c07be4f1-390e-4a00-b99a-85befd69e30b\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " Oct 30 21:19:28.356233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356140 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\" (UniqueName: \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\") pod \"956c8ce3-c567-456d-a73a-2c2971ec4495\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " Oct 30 21:19:28.356233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356196 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\") pod \"735d7add-e2e7-4f2a-be19-2585f3bc731c\" (UID: \"735d7add-e2e7-4f2a-be19-2585f3bc731c\") " Oct 30 21:19:28.356233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356216 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\" (UniqueName: \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\") pod \"c07be4f1-390e-4a00-b99a-85befd69e30b\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " Oct 30 21:19:28.356233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356233 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\") pod \"c07be4f1-390e-4a00-b99a-85befd69e30b\" (UID: \"c07be4f1-390e-4a00-b99a-85befd69e30b\") " Oct 30 21:19:28.356481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.356259 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\") pod \"956c8ce3-c567-456d-a73a-2c2971ec4495\" (UID: \"956c8ce3-c567-456d-a73a-2c2971ec4495\") " Oct 30 21:19:28.357245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.357220 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle" pod "735d7add-e2e7-4f2a-be19-2585f3bc731c" (UID: "735d7add-e2e7-4f2a-be19-2585f3bc731c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.357373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.357344 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle" pod "956c8ce3-c567-456d-a73a-2c2971ec4495" (UID: "956c8ce3-c567-456d-a73a-2c2971ec4495"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.357459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.357430 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle" pod "c07be4f1-390e-4a00-b99a-85befd69e30b" (UID: "c07be4f1-390e-4a00-b99a-85befd69e30b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.358467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.358446 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82" pod "735d7add-e2e7-4f2a-be19-2585f3bc731c" (UID: "735d7add-e2e7-4f2a-be19-2585f3bc731c"). InnerVolumeSpecName "kube-api-access-8ff82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:28.358609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.358593 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc" pod "956c8ce3-c567-456d-a73a-2c2971ec4495" (UID: "956c8ce3-c567-456d-a73a-2c2971ec4495"). InnerVolumeSpecName "kube-api-access-vcztc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:28.359159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.359133 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s" pod "c07be4f1-390e-4a00-b99a-85befd69e30b" (UID: "c07be4f1-390e-4a00-b99a-85befd69e30b"). InnerVolumeSpecName "kube-api-access-zfp8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:28.363453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.363433 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util" pod "956c8ce3-c567-456d-a73a-2c2971ec4495" (UID: "956c8ce3-c567-456d-a73a-2c2971ec4495"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.364956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.364933 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util" pod "735d7add-e2e7-4f2a-be19-2585f3bc731c" (UID: "735d7add-e2e7-4f2a-be19-2585f3bc731c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.365367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.365351 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util" pod "c07be4f1-390e-4a00-b99a-85befd69e30b" (UID: "c07be4f1-390e-4a00-b99a-85befd69e30b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457461 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457482 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcztc\" (UniqueName: \"kubernetes.io/projected/956c8ce3-c567-456d-a73a-2c2971ec4495-kube-api-access-vcztc\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457492 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457500 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp8s\" (UniqueName: \"kubernetes.io/projected/c07be4f1-390e-4a00-b99a-85befd69e30b-kube-api-access-zfp8s\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457508 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c07be4f1-390e-4a00-b99a-85befd69e30b-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.457514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457517 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.458011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457525 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/735d7add-e2e7-4f2a-be19-2585f3bc731c-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.458011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457533 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ff82\" (UniqueName: \"kubernetes.io/projected/735d7add-e2e7-4f2a-be19-2585f3bc731c-kube-api-access-8ff82\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:28.458011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:28.457545 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/956c8ce3-c567-456d-a73a-2c2971ec4495-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:29.178209 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.178177 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" Oct 30 21:19:29.178209 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.178193 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgrrhp" event={"ID":"c07be4f1-390e-4a00-b99a-85befd69e30b","Type":"ContainerDied","Data":"a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa"} Oct 30 21:19:29.178484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.178256 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5dc6108dfcc7cf5d362744215234ac9553e098a35b5005a1515f5fb5be7b1aa" Oct 30 21:19:29.179608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.179582 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:19:29.180357 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.180336 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" Oct 30 21:19:29.180458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.180365 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7" event={"ID":"735d7add-e2e7-4f2a-be19-2585f3bc731c","Type":"ContainerDied","Data":"9c90169ea5da1395c717023e01742afb3576baa6130047a200e86643668295ae"} Oct 30 21:19:29.180458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.180404 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c90169ea5da1395c717023e01742afb3576baa6130047a200e86643668295ae" Oct 30 21:19:29.182395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.182374 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" Oct 30 21:19:29.182395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.182385 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxzrw" event={"ID":"956c8ce3-c567-456d-a73a-2c2971ec4495","Type":"ContainerDied","Data":"925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7"} Oct 30 21:19:29.182533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.182425 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925d1e388fbce3c12b11a0991f1d8abd129afc269db3c1b3fb545382ba3349e7" Oct 30 21:19:29.487195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.487130 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:29.679584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:29.679558 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:31.204377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:31.204341 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:31.497116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:31.497042 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:36.050030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.049994 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h"] Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050420 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050439 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050453 4918 cpu_manager.go:414] "containerMap: removing container" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="extract" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050460 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="extract" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050475 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="util" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050482 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="util" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050491 4918 cpu_manager.go:414] "containerMap: removing container" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050499 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050513 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050520 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="pull" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050530 4918 cpu_manager.go:414] "containerMap: removing container" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="util" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050536 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="util" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050552 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="extract" Oct 30 21:19:36.050555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050559 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050571 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050578 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050586 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050593 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050606 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050613 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050621 4918 cpu_manager.go:414] "containerMap: removing container" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050628 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050637 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050644 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050658 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050664 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050676 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050683 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050693 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050700 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050731 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050738 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050748 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050754 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050769 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050776 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050784 4918 cpu_manager.go:414] "containerMap: removing container" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050792 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050803 4918 cpu_manager.go:414] "containerMap: removing container" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050810 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050820 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050826 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050839 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050846 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050856 4918 cpu_manager.go:414] "containerMap: removing container" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050863 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050874 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050881 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050894 4918 cpu_manager.go:414] "containerMap: removing container" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050901 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="util" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050911 4918 cpu_manager.go:414] "containerMap: removing container" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050918 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="pull" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050928 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.050934 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051018 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="742c6569-8d01-45c5-8096-2ebaa985522f" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051032 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07be4f1-390e-4a00-b99a-85befd69e30b" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051043 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced32944-9c90-4502-8965-6a480dbf30b3" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051052 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e733744d-e7db-4b07-8a60-4dc10dc14b45" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051061 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="735d7add-e2e7-4f2a-be19-2585f3bc731c" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051072 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d514dcd8-4b76-40ac-b323-836fd1e7b15d" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051082 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="956c8ce3-c567-456d-a73a-2c2971ec4495" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051092 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac5100a-6362-4372-9b6a-b039dea7bb09" containerName="extract" Oct 30 21:19:36.051261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.051099 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aecce0-a0ac-4e6d-a355-1207901414b6" containerName="extract" Oct 30 21:19:36.056219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.056200 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.107646 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.107619 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:19:36.109685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.109660 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h"] Oct 30 21:19:36.113570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.113540 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnnp\" (UniqueName: \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.113753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.113726 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.113838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.113784 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.154577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.154551 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:19:36.154683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.154666 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6xs96\"" Oct 30 21:19:36.214451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.214420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.214558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.214461 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.214558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.214492 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnnp\" (UniqueName: \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.214984 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.214960 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.215091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.214992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.273965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.273940 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnnp\" (UniqueName: \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.371873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.371854 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:36.739310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:36.739285 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h"] Oct 30 21:19:36.741317 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:36.741288 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae9be12_0b3a_4c3b_b0ee_bb761d753ab3.slice/crio-9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425 WatchSource:0}: Error finding container 9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425: Status 404 returned error can't find the container with id 9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425 Oct 30 21:19:37.220057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:37.220025 4918 generic.go:356] "Generic (PLEG): container finished" podID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerID="a62917529d7387d1411c6358c14722b71514bf3d5bd29a2aa91fd821898a2743" exitCode=0 Oct 30 21:19:37.220492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:37.220073 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" event={"ID":"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3","Type":"ContainerDied","Data":"a62917529d7387d1411c6358c14722b71514bf3d5bd29a2aa91fd821898a2743"} Oct 30 21:19:37.220492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:37.220110 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" event={"ID":"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3","Type":"ContainerStarted","Data":"9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425"} Oct 30 21:19:38.227198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:38.227163 4918 generic.go:356] "Generic (PLEG): container finished" podID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerID="118c8d29def92fd7015dd6302f618cb9758e9add6925796cf88221e799ce7335" exitCode=0 Oct 30 21:19:38.227566 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:38.227213 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" event={"ID":"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3","Type":"ContainerDied","Data":"118c8d29def92fd7015dd6302f618cb9758e9add6925796cf88221e799ce7335"} Oct 30 21:19:38.235055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:38.235026 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:38.425048 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:38.425021 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:39.061022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.060902 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:39.234453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.234422 4918 generic.go:356] "Generic (PLEG): container finished" podID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerID="54f97303c62218ab1eaeb26b71159f42d18c8a9c220169dd1365a80f991b71f8" exitCode=0 Oct 30 21:19:39.234865 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.234542 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" event={"ID":"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3","Type":"ContainerDied","Data":"54f97303c62218ab1eaeb26b71159f42d18c8a9c220169dd1365a80f991b71f8"} Oct 30 21:19:39.238144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.238120 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk"] Oct 30 21:19:39.241488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.241473 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.291407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291383 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:19:39.291552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291504 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:19:39.291552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291503 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:19:39.291700 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291683 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:19:39.291768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291689 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:19:39.291768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.291690 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5lv4p\"" Oct 30 21:19:39.338429 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338378 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.338429 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338427 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.338561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338495 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.338561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338525 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c76g\" (UniqueName: \"kubernetes.io/projected/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-kube-api-access-6c76g\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.338625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338610 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.338625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.338631 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.359423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.359399 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk"] Oct 30 21:19:39.361916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.361887 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:39.438905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.438883 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439038 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.438910 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439038 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.438928 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c76g\" (UniqueName: \"kubernetes.io/projected/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-kube-api-access-6c76g\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439038 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.438971 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.439056 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.439106 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.439450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.439433 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.441073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.441056 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.487681 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.487660 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:19:39.488050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.488035 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:19:39.492187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.492161 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.492271 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.492245 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.492396 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.492378 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.548023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.548001 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:19:39.597924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.597876 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:19:39.601938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.601915 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c76g\" (UniqueName: \"kubernetes.io/projected/9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4-kube-api-access-6c76g\") pod \"opendatahub-operator-controller-manager-695f8457f8-2wjmk\" (UID: \"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:39.668274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.668252 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:39.848868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.848810 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:39.904872 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.904855 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5lv4p\"" Oct 30 21:19:39.905893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:39.905873 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:40.365418 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.365381 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:40.437994 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:40.437964 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c618e94_72fe_4ce1_bad1_d28ca9f3fcc4.slice/crio-af2153c9499fb043895f5e088e501f99cd88ff7f4633f4402b295600abe30a63 WatchSource:0}: Error finding container af2153c9499fb043895f5e088e501f99cd88ff7f4633f4402b295600abe30a63: Status 404 returned error can't find the container with id af2153c9499fb043895f5e088e501f99cd88ff7f4633f4402b295600abe30a63 Oct 30 21:19:40.446692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.446671 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\") pod \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " Oct 30 21:19:40.446809 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.446699 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\") pod \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " Oct 30 21:19:40.446809 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.446758 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\" (UniqueName: \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\") pod \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\" (UID: \"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3\") " Oct 30 21:19:40.447383 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.447358 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle" pod "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" (UID: "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:40.448804 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.448778 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp" pod "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" (UID: "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3"). InnerVolumeSpecName "kube-api-access-wxnnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:40.452986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.452949 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util" pod "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" (UID: "eae9be12-0b3a-4c3b-b0ee-bb761d753ab3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:40.464334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.464316 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk"] Oct 30 21:19:40.548160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.548111 4918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:40.548160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.548129 4918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-util\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:40.548160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:40.548139 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxnnp\" (UniqueName: \"kubernetes.io/projected/eae9be12-0b3a-4c3b-b0ee-bb761d753ab3-kube-api-access-wxnnp\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:41.245839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:41.245741 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" event={"ID":"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4","Type":"ContainerStarted","Data":"af2153c9499fb043895f5e088e501f99cd88ff7f4633f4402b295600abe30a63"} Oct 30 21:19:41.247731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:41.247690 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" Oct 30 21:19:41.247731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:41.247714 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2kf2h" event={"ID":"eae9be12-0b3a-4c3b-b0ee-bb761d753ab3","Type":"ContainerDied","Data":"9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425"} Oct 30 21:19:41.247901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:41.247762 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5b777b04dc7fabb33cf60f7e95a9e16a21639867c4bedd31751ffe8f20e425" Oct 30 21:19:42.401408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:42.401371 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:43.037590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:43.037560 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:43.928436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:43.928412 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:44.265026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:44.264941 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" event={"ID":"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4","Type":"ContainerStarted","Data":"c35a472024fe11b0a8e1c179ea28c5acce3b4e7041cc3259725381d4444aa009"} Oct 30 21:19:45.271440 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.271408 4918 generic.go:356] "Generic (PLEG): container finished" podID="9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4" containerID="c35a472024fe11b0a8e1c179ea28c5acce3b4e7041cc3259725381d4444aa009" exitCode=0 Oct 30 21:19:45.271851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.271450 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" event={"ID":"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4","Type":"ContainerDied","Data":"c35a472024fe11b0a8e1c179ea28c5acce3b4e7041cc3259725381d4444aa009"} Oct 30 21:19:45.799545 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:19:45.799515 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:19:45.802663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.802639 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh"] Oct 30 21:19:45.803004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.802989 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="pull" Oct 30 21:19:45.803057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803005 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="pull" Oct 30 21:19:45.803057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803017 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="extract" Oct 30 21:19:45.803057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803022 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="extract" Oct 30 21:19:45.803057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803046 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="util" Oct 30 21:19:45.803057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803051 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="util" Oct 30 21:19:45.803206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.803119 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae9be12-0b3a-4c3b-b0ee-bb761d753ab3" containerName="extract" Oct 30 21:19:45.806225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.806209 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:45.864656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.864629 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tbhr2\"" Oct 30 21:19:45.864869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.864846 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:19:45.865239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.864930 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:19:45.865239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.865033 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:19:45.867142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.867120 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh"] Oct 30 21:19:45.890745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.890693 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5s7\" (UniqueName: \"kubernetes.io/projected/3e114750-10c3-400d-92c9-051f0a56cde9-kube-api-access-js5s7\") pod \"dns-operator-controller-manager-8557cbdb84-7h4gh\" (UID: \"3e114750-10c3-400d-92c9-051f0a56cde9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:45.992066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:45.992036 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5s7\" (UniqueName: \"kubernetes.io/projected/3e114750-10c3-400d-92c9-051f0a56cde9-kube-api-access-js5s7\") pod \"dns-operator-controller-manager-8557cbdb84-7h4gh\" (UID: \"3e114750-10c3-400d-92c9-051f0a56cde9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:46.056677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.056616 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5s7\" (UniqueName: \"kubernetes.io/projected/3e114750-10c3-400d-92c9-051f0a56cde9-kube-api-access-js5s7\") pod \"dns-operator-controller-manager-8557cbdb84-7h4gh\" (UID: \"3e114750-10c3-400d-92c9-051f0a56cde9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:46.118176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.118152 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:46.278873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.278840 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" event={"ID":"9c618e94-72fe-4ce1-bad1-d28ca9f3fcc4","Type":"ContainerStarted","Data":"0c55006490081d9f916ea7dd5d89af243eaebc46d0835c07dc01daff4ae6ccf7"} Oct 30 21:19:46.279264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.278996 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:46.414599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.414558 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" podStartSLOduration=2.640657273 podStartE2EDuration="7.414545572s" totalImagesPullingTime="4.357129612s" totalInitContainerRuntime="416.758687ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:46.413531059 +0000 UTC m=+367.107013306" watchObservedRunningTime="2026-10-30 21:19:46.414545572 +0000 UTC m=+367.108027808" Oct 30 21:19:46.618857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.618834 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh"] Oct 30 21:19:46.620975 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:46.620935 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e114750_10c3_400d_92c9_051f0a56cde9.slice/crio-d3ac87e6c345f1260388e7c232bbadcd6e0ad7fac544e7c511d7da92f51c131a WatchSource:0}: Error finding container d3ac87e6c345f1260388e7c232bbadcd6e0ad7fac544e7c511d7da92f51c131a: Status 404 returned error can't find the container with id d3ac87e6c345f1260388e7c232bbadcd6e0ad7fac544e7c511d7da92f51c131a Oct 30 21:19:46.862915 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:46.862880 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:47.285587 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:47.285501 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" event={"ID":"3e114750-10c3-400d-92c9-051f0a56cde9","Type":"ContainerStarted","Data":"d3ac87e6c345f1260388e7c232bbadcd6e0ad7fac544e7c511d7da92f51c131a"} Oct 30 21:19:47.435323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:47.435297 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:48.829080 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:48.829036 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:49.019904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:49.019870 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:50.527195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.527155 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg"] Oct 30 21:19:50.531008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.530983 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:50.628048 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.628000 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9l7s\" (UniqueName: \"kubernetes.io/projected/8629159b-c31f-46e6-a8b4-ec64fbda1493-kube-api-access-r9l7s\") pod \"authorino-operator-7b7bb4bc77-b4hzg\" (UID: \"8629159b-c31f-46e6-a8b4-ec64fbda1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:50.628212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.628173 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hkh8x\"" Oct 30 21:19:50.630905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.630880 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg"] Oct 30 21:19:50.729433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.729403 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9l7s\" (UniqueName: \"kubernetes.io/projected/8629159b-c31f-46e6-a8b4-ec64fbda1493-kube-api-access-r9l7s\") pod \"authorino-operator-7b7bb4bc77-b4hzg\" (UID: \"8629159b-c31f-46e6-a8b4-ec64fbda1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:50.790785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.790725 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9l7s\" (UniqueName: \"kubernetes.io/projected/8629159b-c31f-46e6-a8b4-ec64fbda1493-kube-api-access-r9l7s\") pod \"authorino-operator-7b7bb4bc77-b4hzg\" (UID: \"8629159b-c31f-46e6-a8b4-ec64fbda1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:50.847710 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:50.847684 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:51.212147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.212123 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg"] Oct 30 21:19:51.215247 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:51.215213 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8629159b_c31f_46e6_a8b4_ec64fbda1493.slice/crio-1b6be15a8151956a3180596a9f46143c4ea51495549ded9c95b01cb3e7b7030d WatchSource:0}: Error finding container 1b6be15a8151956a3180596a9f46143c4ea51495549ded9c95b01cb3e7b7030d: Status 404 returned error can't find the container with id 1b6be15a8151956a3180596a9f46143c4ea51495549ded9c95b01cb3e7b7030d Oct 30 21:19:51.308461 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.308427 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" event={"ID":"3e114750-10c3-400d-92c9-051f0a56cde9","Type":"ContainerStarted","Data":"69699b95a153eab8dc4fc09f0632d0cb09fe8d944449e3328fa42a3f950b82ad"} Oct 30 21:19:51.308613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.308560 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:19:51.309815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.309788 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" event={"ID":"8629159b-c31f-46e6-a8b4-ec64fbda1493","Type":"ContainerStarted","Data":"1b6be15a8151956a3180596a9f46143c4ea51495549ded9c95b01cb3e7b7030d"} Oct 30 21:19:51.425605 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.425562 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" podStartSLOduration=2.408961756 podStartE2EDuration="6.425549039s" totalImagesPullingTime="4.016587283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:51.423952987 +0000 UTC m=+372.117435223" watchObservedRunningTime="2026-10-30 21:19:51.425549039 +0000 UTC m=+372.119031275" Oct 30 21:19:51.653957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:51.653896 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:52.004915 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:52.004817 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.026577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:53.026546 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:54.078990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.078958 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:54.326615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.326587 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" event={"ID":"8629159b-c31f-46e6-a8b4-ec64fbda1493","Type":"ContainerStarted","Data":"ab4158eddbb0e94cf201b37526fbe3d7b82e4d8f620874a3b4173e6843170734"} Oct 30 21:19:54.326760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.326740 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:19:54.467238 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.467187 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" podStartSLOduration=1.436727221 podStartE2EDuration="4.46716987s" totalImagesPullingTime="3.030442649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:54.466027948 +0000 UTC m=+375.159510184" watchObservedRunningTime="2026-10-30 21:19:54.46716987 +0000 UTC m=+375.160652107" Oct 30 21:19:54.501045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.501023 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:54.966090 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.966059 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l"] Oct 30 21:19:54.969120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:54.969104 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:55.018430 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.018404 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jk9ds\"" Oct 30 21:19:55.067032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.067009 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l"] Oct 30 21:19:55.067265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.067234 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/7e2402e4-cd7c-4715-b4f0-1a2ab78840f5-kube-api-access-sh4rv\") pod \"limitador-operator-controller-manager-5778bb4449-t252l\" (UID: \"7e2402e4-cd7c-4715-b4f0-1a2ab78840f5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:55.167908 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.167885 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/7e2402e4-cd7c-4715-b4f0-1a2ab78840f5-kube-api-access-sh4rv\") pod \"limitador-operator-controller-manager-5778bb4449-t252l\" (UID: \"7e2402e4-cd7c-4715-b4f0-1a2ab78840f5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:55.232394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.232332 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/7e2402e4-cd7c-4715-b4f0-1a2ab78840f5-kube-api-access-sh4rv\") pod \"limitador-operator-controller-manager-5778bb4449-t252l\" (UID: \"7e2402e4-cd7c-4715-b4f0-1a2ab78840f5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:55.281822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.281795 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:55.598855 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.598764 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:55.657457 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:55.657431 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l"] Oct 30 21:19:55.659494 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:55.659462 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2402e4_cd7c_4715_b4f0_1a2ab78840f5.slice/crio-cc2602bbd915e6a274039c64a5243aec4feb18ceda84ddc2f6321fdd4ac9fa7b WatchSource:0}: Error finding container cc2602bbd915e6a274039c64a5243aec4feb18ceda84ddc2f6321fdd4ac9fa7b: Status 404 returned error can't find the container with id cc2602bbd915e6a274039c64a5243aec4feb18ceda84ddc2f6321fdd4ac9fa7b Oct 30 21:19:56.340874 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:56.340829 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" event={"ID":"7e2402e4-cd7c-4715-b4f0-1a2ab78840f5","Type":"ContainerStarted","Data":"cc2602bbd915e6a274039c64a5243aec4feb18ceda84ddc2f6321fdd4ac9fa7b"} Oct 30 21:19:56.410898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:56.410863 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:56.860297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:56.860267 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:57.213868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.213780 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:57.287755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.287698 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-2wjmk" Oct 30 21:19:57.288319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.288296 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98"] Oct 30 21:19:57.291771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.291752 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.312608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.312585 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98"] Oct 30 21:19:57.385731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.385681 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ctv\" (UniqueName: \"kubernetes.io/projected/401f5991-448c-40a6-9e17-210caf4d6a0c-kube-api-access-b6ctv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.386289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.385793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/401f5991-448c-40a6-9e17-210caf4d6a0c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.386964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.386922 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:19:57.386964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.386953 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:19:57.387169 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.387017 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jg8dg\"" Oct 30 21:19:57.487470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.487396 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ctv\" (UniqueName: \"kubernetes.io/projected/401f5991-448c-40a6-9e17-210caf4d6a0c-kube-api-access-b6ctv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.487611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.487481 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/401f5991-448c-40a6-9e17-210caf4d6a0c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.487923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.487904 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/401f5991-448c-40a6-9e17-210caf4d6a0c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.566198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.566157 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ctv\" (UniqueName: \"kubernetes.io/projected/401f5991-448c-40a6-9e17-210caf4d6a0c-kube-api-access-b6ctv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnb98\" (UID: \"401f5991-448c-40a6-9e17-210caf4d6a0c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:57.605162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:57.605134 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" Oct 30 21:19:58.033530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.033509 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98"] Oct 30 21:19:58.035827 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:19:58.035803 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401f5991_448c_40a6_9e17_210caf4d6a0c.slice/crio-e8c1b5334be6a4b32cca524f85efc722ace744881a7d5e8584fade0e2381e987 WatchSource:0}: Error finding container e8c1b5334be6a4b32cca524f85efc722ace744881a7d5e8584fade0e2381e987: Status 404 returned error can't find the container with id e8c1b5334be6a4b32cca524f85efc722ace744881a7d5e8584fade0e2381e987 Oct 30 21:19:58.354853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.354772 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" event={"ID":"401f5991-448c-40a6-9e17-210caf4d6a0c","Type":"ContainerStarted","Data":"e8c1b5334be6a4b32cca524f85efc722ace744881a7d5e8584fade0e2381e987"} Oct 30 21:19:58.356385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.356346 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" event={"ID":"7e2402e4-cd7c-4715-b4f0-1a2ab78840f5","Type":"ContainerStarted","Data":"b7a18ebe0148bc8989ac8329d237771efb902f81bfaa7904fcab4770da07ddfb"} Oct 30 21:19:58.356604 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.356583 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:19:58.468256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.468208 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" podStartSLOduration=2.10482488 podStartE2EDuration="4.468196507s" totalImagesPullingTime="2.363371627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:58.468001047 +0000 UTC m=+379.161483283" watchObservedRunningTime="2026-10-30 21:19:58.468196507 +0000 UTC m=+379.161678742" Oct 30 21:19:58.751316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.751283 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:58.953538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:58.953504 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:59.246684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:19:59.246649 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:01.376327 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:01.376297 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" event={"ID":"401f5991-448c-40a6-9e17-210caf4d6a0c","Type":"ContainerStarted","Data":"be9bf45b34cab2ed74ee6d007f0544d727eac7431d5a44cedd5b6460b0f42f87"} Oct 30 21:20:01.586385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:01.586334 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnb98" podStartSLOduration=1.727685643 podStartE2EDuration="4.586320762s" totalImagesPullingTime="2.858635119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:01.53831435 +0000 UTC m=+382.231796589" watchObservedRunningTime="2026-10-30 21:20:01.586320762 +0000 UTC m=+382.279802997" Oct 30 21:20:01.993590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:01.993557 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:02.158447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.158416 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:20:02.161994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.161976 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.225609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.225581 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwd4\" (UniqueName: \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.225750 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.225625 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.229296 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.229253 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:02.263979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.263923 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:20:02.276970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.276949 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vbmj6\"" Oct 30 21:20:02.317931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.317912 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-7h4gh" Oct 30 21:20:02.326522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.326501 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwd4\" (UniqueName: \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.326604 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.326545 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.326955 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.326936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.396028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.396002 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwd4\" (UniqueName: \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\") pod \"kuadrant-operator-controller-manager-678d87766f-bxr56\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.474548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.474516 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:02.685025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.684992 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:02.920208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.920173 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:02.973206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:02.973181 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:20:02.975110 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:20:02.975081 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf526fb0_c489_4223_bf45_a5d847151cd3.slice/crio-61bc1d3de8c0f64a24efd43952ba02c10390c1d64fa1f287dde0d2969091a919 WatchSource:0}: Error finding container 61bc1d3de8c0f64a24efd43952ba02c10390c1d64fa1f287dde0d2969091a919: Status 404 returned error can't find the container with id 61bc1d3de8c0f64a24efd43952ba02c10390c1d64fa1f287dde0d2969091a919 Oct 30 21:20:03.386662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:03.386631 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" event={"ID":"cf526fb0-c489-4223-bf45-a5d847151cd3","Type":"ContainerStarted","Data":"61bc1d3de8c0f64a24efd43952ba02c10390c1d64fa1f287dde0d2969091a919"} Oct 30 21:20:04.853627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:04.853594 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:05.039662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:05.039626 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:05.244659 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:05.244615 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:05.337251 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:05.337220 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b4hzg" Oct 30 21:20:05.547268 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:05.547189 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:07.795932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:07.795891 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:08.081621 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:08.081571 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:08.394259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:08.394231 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:08.415449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:08.415419 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" event={"ID":"cf526fb0-c489-4223-bf45-a5d847151cd3","Type":"ContainerStarted","Data":"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e"} Oct 30 21:20:08.415595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:08.415578 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:08.625150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:08.625059 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" podStartSLOduration=1.5258114630000001 podStartE2EDuration="6.625046088s" totalImagesPullingTime="5.099234625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:08.577394752 +0000 UTC m=+389.270876988" watchObservedRunningTime="2026-10-30 21:20:08.625046088 +0000 UTC m=+389.318528323" Oct 30 21:20:09.363968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:09.363940 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-t252l" Oct 30 21:20:10.043625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.043595 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:10.291037 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.291005 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:10.519589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.519558 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:10.589867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.589833 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb"] Oct 30 21:20:10.593556 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.593538 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.663453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.663420 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb"] Oct 30 21:20:10.695168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.695145 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-njg62\"" Oct 30 21:20:10.703482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703459 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjskx\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-kube-api-access-kjskx\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703492 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703515 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703562 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703639 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703666 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703693 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703760 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.703852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.703803 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.725030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.725001 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:10.804375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804346 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804364 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804408 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804429 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804456 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804474 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804502 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.804594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.804568 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjskx\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-kube-api-access-kjskx\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.805231 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.805212 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.805340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.805317 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.805419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.805257 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.805662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.805638 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.805851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.805830 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.807075 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.807049 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.807452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.807429 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.873901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.873872 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjskx\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-kube-api-access-kjskx\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.878027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.878004 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb\" (UID: \"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.905670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.905646 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:10.968920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:10.968828 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:11.326486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.326459 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb"] Oct 30 21:20:11.329142 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:20:11.329099 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8ce2d4_02c2_448c_ad3c_337a0e4bc7e8.slice/crio-d8387469db6499fb8544f99a0929696d7d17b38ce79705f72564e71a06f9545a WatchSource:0}: Error finding container d8387469db6499fb8544f99a0929696d7d17b38ce79705f72564e71a06f9545a: Status 404 returned error can't find the container with id d8387469db6499fb8544f99a0929696d7d17b38ce79705f72564e71a06f9545a Oct 30 21:20:11.331987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.331953 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:20:11.332086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.332021 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:20:11.332086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.332048 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:20:11.433192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.433156 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" event={"ID":"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8","Type":"ContainerStarted","Data":"970f99e4e7114dc4b2123a6a492ac422d31754e4805899379470102b7cb134c1"} Oct 30 21:20:11.433338 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.433207 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" event={"ID":"9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8","Type":"ContainerStarted","Data":"d8387469db6499fb8544f99a0929696d7d17b38ce79705f72564e71a06f9545a"} Oct 30 21:20:11.550385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.550335 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" podStartSLOduration=1.550317771 podStartE2EDuration="1.550317771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:11.550022465 +0000 UTC m=+392.243504713" watchObservedRunningTime="2026-10-30 21:20:11.550317771 +0000 UTC m=+392.243800050" Oct 30 21:20:11.815810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.815773 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:11.906115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:11.906086 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:12.034684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:12.034656 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:12.515228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:12.515199 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:12.848522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:12.848450 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:12.910423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:12.910399 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:13.101446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:13.101354 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:13.442757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:13.442729 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:13.443658 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:13.443642 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb" Oct 30 21:20:13.872116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:13.872085 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:14.058491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:14.058444 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.799817 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:20:15.799779 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:20:17.990013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:17.989983 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:18.639923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:18.639888 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:18.827058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:18.827025 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:19.427576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:19.427550 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:20:22.868042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:22.868007 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s"] Oct 30 21:20:22.873056 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:22.873036 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:22.923013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:22.922990 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:20:22.969575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:22.969551 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:20:22.969792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:22.969775 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-cnkfk\"" Oct 30 21:20:23.008526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.008504 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.008622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.008531 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7djh\" (UniqueName: \"kubernetes.io/projected/12243256-966e-46a5-94e7-f7190dd53c15-kube-api-access-j7djh\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.008622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.008570 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12243256-966e-46a5-94e7-f7190dd53c15-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.008699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.008678 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.064022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.063998 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:20:23.066199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.066179 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s"] Oct 30 21:20:23.109743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.109721 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.109743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.109748 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7djh\" (UniqueName: \"kubernetes.io/projected/12243256-966e-46a5-94e7-f7190dd53c15-kube-api-access-j7djh\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.109882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.109789 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12243256-966e-46a5-94e7-f7190dd53c15-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.109882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.109833 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.110147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.110121 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.110210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.110189 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12243256-966e-46a5-94e7-f7190dd53c15-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.110613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.110596 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12243256-966e-46a5-94e7-f7190dd53c15-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.168063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.168011 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7djh\" (UniqueName: \"kubernetes.io/projected/12243256-966e-46a5-94e7-f7190dd53c15-kube-api-access-j7djh\") pod \"kserve-module-controller-manager-7566cf5fc8-lgj9s\" (UID: \"12243256-966e-46a5-94e7-f7190dd53c15\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.185861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.185841 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:23.600924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:23.600900 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s"] Oct 30 21:20:23.603074 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:20:23.603047 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12243256_966e_46a5_94e7_f7190dd53c15.slice/crio-fd4c43818b1e7df6279aeb85ce965aab22cdf4c4680697fc7fda68a758c43403 WatchSource:0}: Error finding container fd4c43818b1e7df6279aeb85ce965aab22cdf4c4680697fc7fda68a758c43403: Status 404 returned error can't find the container with id fd4c43818b1e7df6279aeb85ce965aab22cdf4c4680697fc7fda68a758c43403 Oct 30 21:20:24.497184 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:24.497140 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" event={"ID":"12243256-966e-46a5-94e7-f7190dd53c15","Type":"ContainerStarted","Data":"fd4c43818b1e7df6279aeb85ce965aab22cdf4c4680697fc7fda68a758c43403"} Oct 30 21:20:25.262154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:25.262087 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:25.449574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:25.449535 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:27.514457 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:27.514418 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" event={"ID":"12243256-966e-46a5-94e7-f7190dd53c15","Type":"ContainerStarted","Data":"da75f8b883917f80db419134dfcc256a44ccf12ddf3ee56183924cd3c86ca3af"} Oct 30 21:20:27.514900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:27.514578 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:20:27.674405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:27.674353 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" podStartSLOduration=2.709863268 podStartE2EDuration="5.674340416s" totalImagesPullingTime="2.964477148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:27.673407484 +0000 UTC m=+408.366889720" watchObservedRunningTime="2026-10-30 21:20:27.674340416 +0000 UTC m=+408.367822651" Oct 30 21:20:30.661451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:30.661417 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:30.851314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:30.851278 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:31.251986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:31.251927 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:31.442300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:31.442262 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:31.620121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:31.620093 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:34.658412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:34.658376 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:34.846168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:34.846134 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:38.660408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:38.660373 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:38.849994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:38.849960 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:39.030981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:39.030902 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:39.226015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:39.225976 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:43.466176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:43.466143 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:43.649385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:43.649354 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:43.832960 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:43.832880 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:45.800901 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:20:45.800865 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:20:48.521815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:20:48.521784 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lgj9s" Oct 30 21:21:15.805364 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:21:15.805329 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:21:28.699073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:28.699036 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:28.902200 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:28.902148 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.093150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.093058 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.311600 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.311570 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.520123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.520078 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.549624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.549588 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wr4p8"] Oct 30 21:21:29.554979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.554951 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.607142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.607112 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:21:29.652273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.652242 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rrt4n\"" Oct 30 21:21:29.654956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.654931 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wr4p8"] Oct 30 21:21:29.728920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.728887 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9k98\" (UniqueName: \"kubernetes.io/projected/9014e08f-c7de-43db-a003-f5513fc49878-kube-api-access-w9k98\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.729264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.728945 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9014e08f-c7de-43db-a003-f5513fc49878-cert\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.736125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.736098 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.829946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.829885 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9k98\" (UniqueName: \"kubernetes.io/projected/9014e08f-c7de-43db-a003-f5513fc49878-kube-api-access-w9k98\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.829946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.829939 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9014e08f-c7de-43db-a003-f5513fc49878-cert\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.832757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.832734 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9014e08f-c7de-43db-a003-f5513fc49878-cert\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.858519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.858493 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5cxn"] Oct 30 21:21:29.862044 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.862024 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:29.887788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.887765 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9k98\" (UniqueName: \"kubernetes.io/projected/9014e08f-c7de-43db-a003-f5513fc49878-kube-api-access-w9k98\") pod \"kserve-controller-manager-776cc66d47-wr4p8\" (UID: \"9014e08f-c7de-43db-a003-f5513fc49878\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:29.911794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.911772 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 21:21:29.912016 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.911994 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-n55wb\"" Oct 30 21:21:29.912119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.912106 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 21:21:29.921302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:29.921281 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5cxn"] Oct 30 21:21:30.031356 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.031331 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-metrics-certs\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.031460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.031364 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwft\" (UniqueName: \"kubernetes.io/projected/6d0b4c70-9500-407d-beb3-e7044ca16689-kube-api-access-8mwft\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.031460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.031387 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-cert\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.131847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.131822 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-metrics-certs\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.131957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.131854 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwft\" (UniqueName: \"kubernetes.io/projected/6d0b4c70-9500-407d-beb3-e7044ca16689-kube-api-access-8mwft\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.131957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.131874 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-cert\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.134254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.134236 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-cert\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.134383 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.134361 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d0b4c70-9500-407d-beb3-e7044ca16689-metrics-certs\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.168257 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.168237 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:30.189617 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.189590 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwft\" (UniqueName: \"kubernetes.io/projected/6d0b4c70-9500-407d-beb3-e7044ca16689-kube-api-access-8mwft\") pod \"odh-model-controller-769875db56-n5cxn\" (UID: \"6d0b4c70-9500-407d-beb3-e7044ca16689\") " pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.475353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.475277 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:30.534015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.533989 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wr4p8"] Oct 30 21:21:30.536560 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:21:30.536530 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9014e08f_c7de_43db_a003_f5513fc49878.slice/crio-f61374f706a93027ab69e697085e5e4a553f87e0b0c5ef16b5b9ba8504823543 WatchSource:0}: Error finding container f61374f706a93027ab69e697085e5e4a553f87e0b0c5ef16b5b9ba8504823543: Status 404 returned error can't find the container with id f61374f706a93027ab69e697085e5e4a553f87e0b0c5ef16b5b9ba8504823543 Oct 30 21:21:30.839200 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.839134 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" event={"ID":"9014e08f-c7de-43db-a003-f5513fc49878","Type":"ContainerStarted","Data":"f61374f706a93027ab69e697085e5e4a553f87e0b0c5ef16b5b9ba8504823543"} Oct 30 21:21:30.847860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.847835 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5cxn"] Oct 30 21:21:30.849983 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:21:30.849958 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b4c70_9500_407d_beb3_e7044ca16689.slice/crio-636bd0e638e49d374ec90fcc57db33f89f8b88c6ba0c63df223444c3074faf39 WatchSource:0}: Error finding container 636bd0e638e49d374ec90fcc57db33f89f8b88c6ba0c63df223444c3074faf39: Status 404 returned error can't find the container with id 636bd0e638e49d374ec90fcc57db33f89f8b88c6ba0c63df223444c3074faf39 Oct 30 21:21:30.889877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:30.889839 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:31.845770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:31.845728 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5cxn" event={"ID":"6d0b4c70-9500-407d-beb3-e7044ca16689","Type":"ContainerStarted","Data":"636bd0e638e49d374ec90fcc57db33f89f8b88c6ba0c63df223444c3074faf39"} Oct 30 21:21:31.884937 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:31.884904 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.862602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:33.862550 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" event={"ID":"9014e08f-c7de-43db-a003-f5513fc49878","Type":"ContainerStarted","Data":"f4759f9813c72f5ea5bab3c170bf38d6af5008e3eec22cb0930bf343d15608e0"} Oct 30 21:21:33.863147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:33.862660 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:21:33.976828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:33.976755 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" podStartSLOduration=2.243228069 podStartE2EDuration="4.976741679s" totalImagesPullingTime="2.73351361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:33.975324408 +0000 UTC m=+474.668806645" watchObservedRunningTime="2026-10-30 21:21:33.976741679 +0000 UTC m=+474.670223915" Oct 30 21:21:34.869096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:34.869061 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5cxn" event={"ID":"6d0b4c70-9500-407d-beb3-e7044ca16689","Type":"ContainerStarted","Data":"69ffa70dc3adde156167e7357cd7d3884fecb2c0d328e0d50760704f711ecbfa"} Oct 30 21:21:34.869518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:34.869145 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:34.981666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:34.981620 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-n5cxn" podStartSLOduration=2.647968901 podStartE2EDuration="5.981604742s" totalImagesPullingTime="3.333635841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:34.980237974 +0000 UTC m=+475.673720210" watchObservedRunningTime="2026-10-30 21:21:34.981604742 +0000 UTC m=+475.675086978" Oct 30 21:21:36.108188 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:36.108153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:36.303190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:36.303153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:36.488177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:36.488144 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:36.675655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:36.675620 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:36.860294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:36.860255 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:37.043494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:37.043466 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:37.243978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:37.243905 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:45.800017 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:21:45.799983 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:21:45.876642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:45.876618 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-n5cxn" Oct 30 21:21:49.959969 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:49.959937 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.145819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:50.145779 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:52.516902 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:52.516866 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:52.711812 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:21:52.711786 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:04.874906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:22:04.874877 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wr4p8" Oct 30 21:22:15.800229 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:22:15.800193 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:22:32.221962 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:22:32.221928 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:32.406458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:22:32.406410 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:45.799548 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:22:45.799515 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:23:15.800731 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:23:15.800606 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:23:25.687233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:25.687158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:32.537363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:32.537330 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:39.457723 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.457680 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:23:39.460459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.460436 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:23:39.470568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.470541 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:23:39.472928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.472908 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:23:39.475540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.475522 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:23:39.477852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:23:39.477835 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:23:45.799779 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:23:45.799740 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:24:15.799388 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:24:15.799353 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:24:19.721122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:24:19.721067 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:21.236258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:24:21.236223 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:32.885675 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:24:32.885637 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:40.517797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:24:40.517735 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:45.799723 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:24:45.799665 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:25:15.799723 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:25:15.799680 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:25:33.235906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:25:33.235870 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:33.431256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:25:33.431224 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:45.800034 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:25:45.800000 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:26:09.006254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:26:09.006204 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:09.248095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:26:09.248018 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:15.799790 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:26:15.799753 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:26:33.583407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:26:33.583301 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:36.447332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:26:36.447289 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:36.639910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:26:36.639879 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.800449 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:26:45.800410 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:27:15.799860 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:27:15.799821 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:27:33.916441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:27:33.916406 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:45.800692 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:27:45.800654 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:28:15.799747 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:28:15.799696 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:28:30.356433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:30.356393 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:34.400369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:34.400327 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:34.594345 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:34.594310 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:38.862924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:38.862865 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:39.052624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.052589 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:39.492909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.492882 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:28:39.496754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.496726 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:28:39.510236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.510214 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:28:39.513726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.513686 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:28:39.515599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.515580 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:28:39.518749 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:39.518732 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:28:45.799139 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:28:45.799101 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:28:47.059744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:28:47.059700 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:15.800478 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:29:15.800445 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:29:34.784670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:29:34.784600 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:36.998382 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:29:36.998344 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:45.800502 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:29:45.800453 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:30:15.800208 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:30:15.800171 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:30:28.771042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:28.771010 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bxr56_cf526fb0-c489-4223-bf45-a5d847151cd3/manager/0.log" Oct 30 21:30:29.146254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.146220 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:30:29.146488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.146454 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" podUID="cf526fb0-c489-4223-bf45-a5d847151cd3" containerName="manager" containerID="cri-o://173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e" gracePeriod=10 Oct 30 21:30:29.148456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.148435 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 21:30:29.152006 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.151989 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.255657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.255632 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.255778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.255679 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjgc\" (UniqueName: \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.272022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.271997 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 21:30:29.356655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.356615 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.356791 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.356696 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjgc\" (UniqueName: \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.356998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.356979 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.404906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.404855 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:30:29.423512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.423484 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjgc\" (UniqueName: \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\") pod \"kuadrant-operator-controller-manager-678d87766f-l6x6k\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.540534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.540498 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:29.558158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.558130 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\" (UniqueName: \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\") pod \"cf526fb0-c489-4223-bf45-a5d847151cd3\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " Oct 30 21:30:29.558305 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.558286 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\") pod \"cf526fb0-c489-4223-bf45-a5d847151cd3\" (UID: \"cf526fb0-c489-4223-bf45-a5d847151cd3\") " Oct 30 21:30:29.558936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.558884 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume" pod "cf526fb0-c489-4223-bf45-a5d847151cd3" (UID: "cf526fb0-c489-4223-bf45-a5d847151cd3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:29.561167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.561140 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4" pod "cf526fb0-c489-4223-bf45-a5d847151cd3" (UID: "cf526fb0-c489-4223-bf45-a5d847151cd3"). InnerVolumeSpecName "kube-api-access-mjwd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:29.639879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.639846 4918 generic.go:356] "Generic (PLEG): container finished" podID="cf526fb0-c489-4223-bf45-a5d847151cd3" containerID="173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e" exitCode=0 Oct 30 21:30:29.640025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.639902 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" Oct 30 21:30:29.640025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.639900 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" event={"ID":"cf526fb0-c489-4223-bf45-a5d847151cd3","Type":"ContainerDied","Data":"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e"} Oct 30 21:30:29.640128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.640044 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56" event={"ID":"cf526fb0-c489-4223-bf45-a5d847151cd3","Type":"ContainerDied","Data":"61bc1d3de8c0f64a24efd43952ba02c10390c1d64fa1f287dde0d2969091a919"} Oct 30 21:30:29.640128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.640079 4918 scope.go:122] "RemoveContainer" containerID="173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e" Oct 30 21:30:29.649247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.649214 4918 scope.go:122] "RemoveContainer" containerID="173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e" Oct 30 21:30:29.649485 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:30:29.649466 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e\": container with ID starting with 173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e not found: ID does not exist" containerID="173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e" Oct 30 21:30:29.649546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.649493 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e"} err="failed to get container status \"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e\": rpc error: code = NotFound desc = could not find container \"173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e\": container with ID starting with 173e2b5cc59a464d3ca1752ec5942436f6a1752496e6f7635ba87fac8da60b7e not found: ID does not exist" Oct 30 21:30:29.659967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.659920 4918 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf526fb0-c489-4223-bf45-a5d847151cd3-extensions-socket-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:29.659967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.659942 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwd4\" (UniqueName: \"kubernetes.io/projected/cf526fb0-c489-4223-bf45-a5d847151cd3-kube-api-access-mjwd4\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:29.809723 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.809686 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:30:29.815853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.815828 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxr56"] Oct 30 21:30:29.969252 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:30:29.969225 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2f906f_cf82_44c1_a32c_ace501143a05.slice/crio-1ec3ebd0ddb473d5fb356962253b9d94c0605f38f2c712d2a23e6be5e269b36d WatchSource:0}: Error finding container 1ec3ebd0ddb473d5fb356962253b9d94c0605f38f2c712d2a23e6be5e269b36d: Status 404 returned error can't find the container with id 1ec3ebd0ddb473d5fb356962253b9d94c0605f38f2c712d2a23e6be5e269b36d Oct 30 21:30:29.971827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:29.971802 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 21:30:30.649568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:30.649529 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" event={"ID":"db2f906f-cf82-44c1-a32c-ace501143a05","Type":"ContainerStarted","Data":"6e8273a2240c089b6f837421a0c5a1eac820e8e27f6bbc7d158d4442e21b9ec0"} Oct 30 21:30:30.649751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:30.649677 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:30.649751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:30.649724 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" event={"ID":"db2f906f-cf82-44c1-a32c-ace501143a05","Type":"ContainerStarted","Data":"1ec3ebd0ddb473d5fb356962253b9d94c0605f38f2c712d2a23e6be5e269b36d"} Oct 30 21:30:30.774529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:30.774481 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" podStartSLOduration=1.774469722 podStartE2EDuration="1.774469722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:30.773257248 +0000 UTC m=+1011.466739486" watchObservedRunningTime="2026-10-30 21:30:30.774469722 +0000 UTC m=+1011.467951958" Oct 30 21:30:31.426801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:31.426766 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf526fb0-c489-4223-bf45-a5d847151cd3" path="/var/lib/kubelet/pods/cf526fb0-c489-4223-bf45-a5d847151cd3/volumes" Oct 30 21:30:34.940226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:34.940194 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:41.657557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:41.657521 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 21:30:45.799683 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:30:45.799645 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:30:51.679440 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.679403 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:51.698423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.698358 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:30:51.699742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.699079 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cf526fb0-c489-4223-bf45-a5d847151cd3" containerName="manager" Oct 30 21:30:51.699742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.699105 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf526fb0-c489-4223-bf45-a5d847151cd3" containerName="manager" Oct 30 21:30:51.699742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.699223 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf526fb0-c489-4223-bf45-a5d847151cd3" containerName="manager" Oct 30 21:30:51.704213 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.704193 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.727684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.727660 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.727811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.727733 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt98\" (UniqueName: \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.763392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.763365 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:30:51.814317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.814294 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b945v\"" Oct 30 21:30:51.816460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.816443 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:30:51.828480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.828460 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.828572 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.828503 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt98\" (UniqueName: \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.829304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.829282 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.874571 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.874537 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:51.890405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.890379 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt98\" (UniqueName: \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\") pod \"limitador-limitador-74dfcfb894-c7jcq\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:51.938659 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:51.938581 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:30:52.017972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.017944 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:52.111082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.111053 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:52.300388 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.300303 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:52.386312 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:30:52.386280 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2477411a_6123_40ce_a161_08646fd3c43b.slice/crio-4dcc8f784067cf579a80cb1f8b54d81924a8abd3fb8bbbdd09262cdad3d0c388 WatchSource:0}: Error finding container 4dcc8f784067cf579a80cb1f8b54d81924a8abd3fb8bbbdd09262cdad3d0c388: Status 404 returned error can't find the container with id 4dcc8f784067cf579a80cb1f8b54d81924a8abd3fb8bbbdd09262cdad3d0c388 Oct 30 21:30:52.388904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.388880 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:30:52.389087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.389060 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:30:52.762717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:52.762673 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" event={"ID":"2477411a-6123-40ce-a161-08646fd3c43b","Type":"ContainerStarted","Data":"4dcc8f784067cf579a80cb1f8b54d81924a8abd3fb8bbbdd09262cdad3d0c388"} Oct 30 21:30:53.437190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:53.437154 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:53.626851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:53.626819 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:55.779677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:55.779637 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" event={"ID":"2477411a-6123-40ce-a161-08646fd3c43b","Type":"ContainerStarted","Data":"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3"} Oct 30 21:30:55.780104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:55.779781 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:30:55.900526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:30:55.900479 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" podStartSLOduration=2.393429732 podStartE2EDuration="4.900466299s" totalImagesPullingTime="2.507036567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:55.898417852 +0000 UTC m=+1036.591900088" watchObservedRunningTime="2026-10-30 21:30:55.900466299 +0000 UTC m=+1036.593948645" Oct 30 21:31:06.786225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:06.786193 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:31:10.190264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.190171 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:31:10.190740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.190476 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" podUID="2477411a-6123-40ce-a161-08646fd3c43b" containerName="limitador" containerID="cri-o://b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3" gracePeriod=30 Oct 30 21:31:10.733214 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.733191 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:31:10.787743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.787688 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\" (UniqueName: \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\") pod \"2477411a-6123-40ce-a161-08646fd3c43b\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " Oct 30 21:31:10.787850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.787766 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\" (UniqueName: \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\") pod \"2477411a-6123-40ce-a161-08646fd3c43b\" (UID: \"2477411a-6123-40ce-a161-08646fd3c43b\") " Oct 30 21:31:10.788045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.788020 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file" pod "2477411a-6123-40ce-a161-08646fd3c43b" (UID: "2477411a-6123-40ce-a161-08646fd3c43b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:10.789713 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.789678 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98" pod "2477411a-6123-40ce-a161-08646fd3c43b" (UID: "2477411a-6123-40ce-a161-08646fd3c43b"). InnerVolumeSpecName "kube-api-access-ptt98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:10.859880 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.859847 4918 generic.go:356] "Generic (PLEG): container finished" podID="2477411a-6123-40ce-a161-08646fd3c43b" containerID="b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3" exitCode=0 Oct 30 21:31:10.859997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.859888 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" Oct 30 21:31:10.859997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.859900 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" event={"ID":"2477411a-6123-40ce-a161-08646fd3c43b","Type":"ContainerDied","Data":"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3"} Oct 30 21:31:10.859997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.859968 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c7jcq" event={"ID":"2477411a-6123-40ce-a161-08646fd3c43b","Type":"ContainerDied","Data":"4dcc8f784067cf579a80cb1f8b54d81924a8abd3fb8bbbdd09262cdad3d0c388"} Oct 30 21:31:10.860122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.860003 4918 scope.go:122] "RemoveContainer" containerID="b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3" Oct 30 21:31:10.870611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.870591 4918 scope.go:122] "RemoveContainer" containerID="b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3" Oct 30 21:31:10.870937 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:10.870909 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3\": container with ID starting with b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3 not found: ID does not exist" containerID="b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3" Oct 30 21:31:10.871042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.870948 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3"} err="failed to get container status \"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3\": rpc error: code = NotFound desc = could not find container \"b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3\": container with ID starting with b8a8a3ef3365c05dc3b33d20849a98b57514081b091d7bac9184e2c41c620ba3 not found: ID does not exist" Oct 30 21:31:10.889060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.889033 4918 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2477411a-6123-40ce-a161-08646fd3c43b-config-file\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:10.889060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:10.889058 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt98\" (UniqueName: \"kubernetes.io/projected/2477411a-6123-40ce-a161-08646fd3c43b-kube-api-access-ptt98\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:11.027479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:11.027453 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:31:11.031222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:11.031203 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c7jcq"] Oct 30 21:31:11.426779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:11.426744 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2477411a-6123-40ce-a161-08646fd3c43b" path="/var/lib/kubelet/pods/2477411a-6123-40ce-a161-08646fd3c43b/volumes" Oct 30 21:31:14.006993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.006963 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:14.221596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.221559 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:14.711799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.711758 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z"] Oct 30 21:31:14.712165 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.712150 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2477411a-6123-40ce-a161-08646fd3c43b" containerName="limitador" Oct 30 21:31:14.712235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.712165 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2477411a-6123-40ce-a161-08646fd3c43b" containerName="limitador" Oct 30 21:31:14.712292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.712237 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2477411a-6123-40ce-a161-08646fd3c43b" containerName="limitador" Oct 30 21:31:14.716769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.716748 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.770662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.770532 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z"] Oct 30 21:31:14.814932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.814895 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/672b478c-ed93-4fce-921e-70ea550d0b20-data\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.815070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.815007 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vmk\" (UniqueName: \"kubernetes.io/projected/672b478c-ed93-4fce-921e-70ea550d0b20-kube-api-access-x4vmk\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.818641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.818620 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:31:14.818641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.818632 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:31:14.818802 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.818649 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:31:14.821207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.821186 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-btgzd\"" Oct 30 21:31:14.855775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.855751 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:14.916069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.916035 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vmk\" (UniqueName: \"kubernetes.io/projected/672b478c-ed93-4fce-921e-70ea550d0b20-kube-api-access-x4vmk\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.916229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.916075 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/672b478c-ed93-4fce-921e-70ea550d0b20-data\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.916464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.916443 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/672b478c-ed93-4fce-921e-70ea550d0b20-data\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:14.978400 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:14.978342 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vmk\" (UniqueName: \"kubernetes.io/projected/672b478c-ed93-4fce-921e-70ea550d0b20-kube-api-access-x4vmk\") pod \"postgres-7cdbf6dc44-4kd2z\" (UID: \"672b478c-ed93-4fce-921e-70ea550d0b20\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:15.029524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:15.029502 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:15.063689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:15.063664 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:15.246810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:15.246728 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:15.401101 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:31:15.401074 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672b478c_ed93_4fce_921e_70ea550d0b20.slice/crio-1068985cb5d07bab35128be03661dc0a462dbbd82cf4d49ce4b80c5e339eefd5 WatchSource:0}: Error finding container 1068985cb5d07bab35128be03661dc0a462dbbd82cf4d49ce4b80c5e339eefd5: Status 404 returned error can't find the container with id 1068985cb5d07bab35128be03661dc0a462dbbd82cf4d49ce4b80c5e339eefd5 Oct 30 21:31:15.403656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:15.403636 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z"] Oct 30 21:31:15.799370 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:15.799339 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:31:15.888533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:15.888501 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" event={"ID":"672b478c-ed93-4fce-921e-70ea550d0b20","Type":"ContainerStarted","Data":"1068985cb5d07bab35128be03661dc0a462dbbd82cf4d49ce4b80c5e339eefd5"} Oct 30 21:31:22.931429 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:22.931396 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" event={"ID":"672b478c-ed93-4fce-921e-70ea550d0b20","Type":"ContainerStarted","Data":"87217d5facedc84d0c3487a717a3eebcb4a0cfedae92402fa86008a0ce6a5188"} Oct 30 21:31:22.931928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:22.931532 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:23.049421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:23.049378 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" podStartSLOduration=1.954011635 podStartE2EDuration="9.049365631s" totalImagesPullingTime="7.095353996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:23.048684388 +0000 UTC m=+1063.742166623" watchObservedRunningTime="2026-10-30 21:31:23.049365631 +0000 UTC m=+1063.742847867" Oct 30 21:31:28.966094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:28.966062 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4kd2z" Oct 30 21:31:35.291044 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:35.290997 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.228360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.228331 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.256050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.256025 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:36.260482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.260466 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.314469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.314444 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:31:36.314930 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.314452 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:31:36.315051 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.314512 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:31:36.317162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.317140 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:36.365052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.365030 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5nx5r\"" Oct 30 21:31:36.397265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.397236 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.397375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.397320 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.397375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.397351 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9l2\" (UniqueName: \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.459803 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.459780 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.498397 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.498345 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.498476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.498400 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.498476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.498419 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9l2\" (UniqueName: \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.498545 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:36.498476 4918 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Oct 30 21:31:36.498545 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:36.498529 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs podName:90a23c63-2564-4d0f-a670-216fab6f012a nodeName:}" failed. No retries permitted until 2026-10-30 21:31:36.998514886 +0000 UTC m=+1077.691997105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs") pod "maas-controller-579cf7c6d-vzfnf" (UID: "90a23c63-2564-4d0f-a670-216fab6f012a") : secret "maas-controller-metrics-tls" not found Oct 30 21:31:36.500848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.500830 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.561013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.560983 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9l2\" (UniqueName: \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:36.673062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.673017 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.824451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.824373 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 21:31:36.828672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.828651 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:36.871560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.871524 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.885087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:36.885066 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 21:31:37.002848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.002821 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.002996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.002871 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.002996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.002949 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:37.002996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.002988 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cx9\" (UniqueName: \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.005245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.005222 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") pod \"maas-controller-579cf7c6d-vzfnf\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:37.103828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.103778 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cx9\" (UniqueName: \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.103828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.103821 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.103945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.103855 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.106053 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.106033 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.106132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.106073 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.165166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.165143 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cx9\" (UniqueName: \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\") pod \"maas-controller-744f599cc5-hxk6h\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.173963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.173941 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:37.442133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.442105 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:37.544264 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:31:37.544233 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a23c63_2564_4d0f_a670_216fab6f012a.slice/crio-5a6573e668ecdcf425d6b74661f910727617cf566f10a760da9ee42b7fbe4958 WatchSource:0}: Error finding container 5a6573e668ecdcf425d6b74661f910727617cf566f10a760da9ee42b7fbe4958: Status 404 returned error can't find the container with id 5a6573e668ecdcf425d6b74661f910727617cf566f10a760da9ee42b7fbe4958 Oct 30 21:31:37.547054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.547027 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:37.809725 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:31:37.809687 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c8f68_3fb9_4a35_a187_1ef25cf0617c.slice/crio-0c7435af9cedaa0cbd156dcbf966a61f62fcc071afcfff604388fa2b2a7d0ebe WatchSource:0}: Error finding container 0c7435af9cedaa0cbd156dcbf966a61f62fcc071afcfff604388fa2b2a7d0ebe: Status 404 returned error can't find the container with id 0c7435af9cedaa0cbd156dcbf966a61f62fcc071afcfff604388fa2b2a7d0ebe Oct 30 21:31:37.811401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:37.811380 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 21:31:38.009543 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:38.009507 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744f599cc5-hxk6h" event={"ID":"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c","Type":"ContainerStarted","Data":"0c7435af9cedaa0cbd156dcbf966a61f62fcc071afcfff604388fa2b2a7d0ebe"} Oct 30 21:31:38.010649 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:38.010622 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" event={"ID":"90a23c63-2564-4d0f-a670-216fab6f012a","Type":"ContainerStarted","Data":"5a6573e668ecdcf425d6b74661f910727617cf566f10a760da9ee42b7fbe4958"} Oct 30 21:31:41.005373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:41.005342 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:31:41.437965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:41.437932 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.625222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:41.625189 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:42.037930 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.037843 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" event={"ID":"90a23c63-2564-4d0f-a670-216fab6f012a","Type":"ContainerStarted","Data":"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0"} Oct 30 21:31:42.038351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.038036 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:42.039474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.039448 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744f599cc5-hxk6h" event={"ID":"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c","Type":"ContainerStarted","Data":"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf"} Oct 30 21:31:42.039577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.039501 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:42.159701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.159658 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" podStartSLOduration=2.753474626 podStartE2EDuration="6.159644929s" totalImagesPullingTime="3.406170303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:42.1577575 +0000 UTC m=+1082.851239736" watchObservedRunningTime="2026-10-30 21:31:42.159644929 +0000 UTC m=+1082.853127164" Oct 30 21:31:42.280808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:42.280751 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-744f599cc5-hxk6h" podStartSLOduration=2.861853249 podStartE2EDuration="6.280695328s" totalImagesPullingTime="3.418842079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:42.278386818 +0000 UTC m=+1082.971869055" watchObservedRunningTime="2026-10-30 21:31:42.280695328 +0000 UTC m=+1082.974177565" Oct 30 21:31:44.608639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.608607 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp"] Oct 30 21:31:44.612470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.612432 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.666487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.666458 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.666634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.666521 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.666634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.666554 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.666634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.666618 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6r2\" (UniqueName: \"kubernetes.io/projected/27fd99b3-1c94-498b-86f9-7c01e5b8576c-kube-api-access-jj6r2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.668921 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.668900 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp"] Oct 30 21:31:44.717015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.716977 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 21:31:44.717181 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.717159 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 21:31:44.717249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.717167 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 21:31:44.717249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.717189 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-s48zq\"" Oct 30 21:31:44.767076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.767051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.767220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.767092 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.767220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.767154 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6r2\" (UniqueName: \"kubernetes.io/projected/27fd99b3-1c94-498b-86f9-7c01e5b8576c-kube-api-access-jj6r2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.767220 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:44.767197 4918 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 21:31:44.767372 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:44.767223 4918 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 21:31:44.767372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.767236 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.767372 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:44.767261 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert podName:27fd99b3-1c94-498b-86f9-7c01e5b8576c nodeName:}" failed. No retries permitted until 2026-10-30 21:31:45.267241854 +0000 UTC m=+1085.960724093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" (UID: "27fd99b3-1c94-498b-86f9-7c01e5b8576c") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 21:31:44.767372 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:44.767355 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source podName:27fd99b3-1c94-498b-86f9-7c01e5b8576c nodeName:}" failed. No retries permitted until 2026-10-30 21:31:45.267328051 +0000 UTC m=+1085.960810270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" (UID: "27fd99b3-1c94-498b-86f9-7c01e5b8576c") : configmap "usage-tenancy-proxy-source" not found Oct 30 21:31:44.767903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.767882 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:44.804147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.804112 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.829419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:44.829394 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6r2\" (UniqueName: \"kubernetes.io/projected/27fd99b3-1c94-498b-86f9-7c01e5b8576c-kube-api-access-jj6r2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.156230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.156198 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:45.271143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.271106 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.271313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.271162 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.272187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.272163 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/27fd99b3-1c94-498b-86f9-7c01e5b8576c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.273541 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.273521 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27fd99b3-1c94-498b-86f9-7c01e5b8576c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzxp\" (UID: \"27fd99b3-1c94-498b-86f9-7c01e5b8576c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.525847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.525764 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:45.801777 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:45.801685 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:31:45.898778 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:31:45.898739 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd99b3_1c94_498b_86f9_7c01e5b8576c.slice/crio-1a17e39eb15e234dc48df506cd5d644a20acd3fb839fc9f820ef5268b2456011 WatchSource:0}: Error finding container 1a17e39eb15e234dc48df506cd5d644a20acd3fb839fc9f820ef5268b2456011: Status 404 returned error can't find the container with id 1a17e39eb15e234dc48df506cd5d644a20acd3fb839fc9f820ef5268b2456011 Oct 30 21:31:45.901773 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:45.901754 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp"] Oct 30 21:31:46.071744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:46.071652 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" event={"ID":"27fd99b3-1c94-498b-86f9-7c01e5b8576c","Type":"ContainerStarted","Data":"1a17e39eb15e234dc48df506cd5d644a20acd3fb839fc9f820ef5268b2456011"} Oct 30 21:31:53.053621 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:53.053590 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:53.054134 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:53.053671 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 21:31:53.327340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:53.327240 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:53.327554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:53.327526 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" podUID="90a23c63-2564-4d0f-a670-216fab6f012a" containerName="manager" containerID="cri-o://0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0" gracePeriod=30 Oct 30 21:31:54.105163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.105136 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:54.124105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.124076 4918 generic.go:356] "Generic (PLEG): container finished" podID="90a23c63-2564-4d0f-a670-216fab6f012a" containerID="0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0" exitCode=0 Oct 30 21:31:54.124247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.124124 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" event={"ID":"90a23c63-2564-4d0f-a670-216fab6f012a","Type":"ContainerDied","Data":"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0"} Oct 30 21:31:54.124247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.124127 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" Oct 30 21:31:54.124247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.124207 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vzfnf" event={"ID":"90a23c63-2564-4d0f-a670-216fab6f012a","Type":"ContainerDied","Data":"5a6573e668ecdcf425d6b74661f910727617cf566f10a760da9ee42b7fbe4958"} Oct 30 21:31:54.124247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.124229 4918 scope.go:122] "RemoveContainer" containerID="0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0" Oct 30 21:31:54.133882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.133863 4918 scope.go:122] "RemoveContainer" containerID="0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0" Oct 30 21:31:54.134272 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:31:54.134209 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0\": container with ID starting with 0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0 not found: ID does not exist" containerID="0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0" Oct 30 21:31:54.134272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.134246 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0"} err="failed to get container status \"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0\": rpc error: code = NotFound desc = could not find container \"0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0\": container with ID starting with 0ad3b1df0b5ffbbf7712e626132fdbffc9b409161723f2a69dd0529381d8a9e0 not found: ID does not exist" Oct 30 21:31:54.152146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.152124 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") pod \"90a23c63-2564-4d0f-a670-216fab6f012a\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " Oct 30 21:31:54.152253 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.152212 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\" (UniqueName: \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\") pod \"90a23c63-2564-4d0f-a670-216fab6f012a\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " Oct 30 21:31:54.152308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.152283 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\") pod \"90a23c63-2564-4d0f-a670-216fab6f012a\" (UID: \"90a23c63-2564-4d0f-a670-216fab6f012a\") " Oct 30 21:31:54.154609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.154581 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs" pod "90a23c63-2564-4d0f-a670-216fab6f012a" (UID: "90a23c63-2564-4d0f-a670-216fab6f012a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:54.154733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.154653 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert" pod "90a23c63-2564-4d0f-a670-216fab6f012a" (UID: "90a23c63-2564-4d0f-a670-216fab6f012a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:54.154733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.154647 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2" pod "90a23c63-2564-4d0f-a670-216fab6f012a" (UID: "90a23c63-2564-4d0f-a670-216fab6f012a"). InnerVolumeSpecName "kube-api-access-xt9l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:54.253591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.253509 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt9l2\" (UniqueName: \"kubernetes.io/projected/90a23c63-2564-4d0f-a670-216fab6f012a-kube-api-access-xt9l2\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:54.253591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.253546 4918 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-webhook-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:54.253591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.253563 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a23c63-2564-4d0f-a670-216fab6f012a-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:54.550650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.550573 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:54.554636 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:54.554607 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vzfnf"] Oct 30 21:31:55.427375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:55.427342 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a23c63-2564-4d0f-a670-216fab6f012a" path="/var/lib/kubelet/pods/90a23c63-2564-4d0f-a670-216fab6f012a/volumes" Oct 30 21:31:59.158816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:59.158783 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" event={"ID":"27fd99b3-1c94-498b-86f9-7c01e5b8576c","Type":"ContainerStarted","Data":"f5f246d41dd314f5fc81e40097d4aad368ae47300716c87da6e5ed2251c26cd1"} Oct 30 21:31:59.159221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:59.158868 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:31:59.279190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:31:59.279142 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" podStartSLOduration=2.350448685 podStartE2EDuration="15.279127028s" totalImagesPullingTime="12.928678343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:59.2772303 +0000 UTC m=+1099.970712536" watchObservedRunningTime="2026-10-30 21:31:59.279127028 +0000 UTC m=+1099.972609264" Oct 30 21:32:10.169813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:10.169700 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzxp" Oct 30 21:32:10.772623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:10.772589 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:15.800250 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:32:15.800217 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:32:17.360525 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:17.360490 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:17.583655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:17.583621 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:19.505085 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:19.505031 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:19.709016 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:19.708938 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:19.920777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:19.920742 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.516404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.516371 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:20.516788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.516761 4918 cpu_manager.go:414] "containerMap: removing container" podUID="90a23c63-2564-4d0f-a670-216fab6f012a" containerName="manager" Oct 30 21:32:20.516788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.516774 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a23c63-2564-4d0f-a670-216fab6f012a" containerName="manager" Oct 30 21:32:20.516860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.516849 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a23c63-2564-4d0f-a670-216fab6f012a" containerName="manager" Oct 30 21:32:20.522721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.522693 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.578285 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.578258 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:20.626168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.626146 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hkklw\"" Oct 30 21:32:20.626168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.626167 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:32:20.627283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.627259 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:32:20.666030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.666002 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twp5w\" (UniqueName: \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.666139 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.666073 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.666202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.666146 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.666261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.666206 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.767023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.766970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.767023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.767012 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.767182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.767033 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.767182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.767068 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twp5w\" (UniqueName: \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.767698 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.767680 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.769342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.769321 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.769431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.769367 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.832869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.832843 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twp5w\" (UniqueName: \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\") pod \"maas-api-6d669c7fcd-2967z\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:20.835661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:20.835640 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:21.215667 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:32:21.215637 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146a917a_de8e_4804_a5d0_229ee8c1bead.slice/crio-7e0084293a1da0af91dc1971041c7bfc9939bf430f8923a901f8ac5daafbf739 WatchSource:0}: Error finding container 7e0084293a1da0af91dc1971041c7bfc9939bf430f8923a901f8ac5daafbf739: Status 404 returned error can't find the container with id 7e0084293a1da0af91dc1971041c7bfc9939bf430f8923a901f8ac5daafbf739 Oct 30 21:32:21.218188 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:21.218158 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:21.272581 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:21.272553 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" event={"ID":"146a917a-de8e-4804-a5d0-229ee8c1bead","Type":"ContainerStarted","Data":"7e0084293a1da0af91dc1971041c7bfc9939bf430f8923a901f8ac5daafbf739"} Oct 30 21:32:24.289529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:24.289498 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" event={"ID":"146a917a-de8e-4804-a5d0-229ee8c1bead","Type":"ContainerStarted","Data":"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab"} Oct 30 21:32:24.289992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:24.289558 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:24.416254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:24.416211 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" podStartSLOduration=2.360765382 podStartE2EDuration="4.416196763s" totalImagesPullingTime="2.055431381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:24.4150428 +0000 UTC m=+1125.108525035" watchObservedRunningTime="2026-10-30 21:32:24.416196763 +0000 UTC m=+1125.109678999" Oct 30 21:32:30.302117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:30.302082 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:30.921358 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:30.921319 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:31.160606 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:31.160575 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:34.842537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:34.841897 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:55686->10.132.0.34:15021: read: connection reset by peer" start-of-body= Oct 30 21:32:34.842537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:34.841966 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:55686->10.132.0.34:15021: read: connection reset by peer" Oct 30 21:32:35.353852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.353820 4918 generic.go:356] "Generic (PLEG): container finished" podID="aef085da-a319-4c09-a236-af0467dc72ec" containerID="e6c0a7f8f2bac818adcbb861313ee3eaf43070ab383b41e85fad8119f6695023" exitCode=0 Oct 30 21:32:35.354020 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.353884 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" event={"ID":"aef085da-a319-4c09-a236-af0467dc72ec","Type":"ContainerDied","Data":"e6c0a7f8f2bac818adcbb861313ee3eaf43070ab383b41e85fad8119f6695023"} Oct 30 21:32:35.354294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.354270 4918 scope.go:122] "RemoveContainer" containerID="e6c0a7f8f2bac818adcbb861313ee3eaf43070ab383b41e85fad8119f6695023" Oct 30 21:32:35.355373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.355347 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:32:35.355437 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.355417 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:32:35.355477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.355466 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:32:35.638474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:35.637498 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:36.361214 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:36.361179 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" event={"ID":"aef085da-a319-4c09-a236-af0467dc72ec","Type":"ContainerStarted","Data":"e68a4d40da640d0494885e4b1bc904c05f8790e7af51f4dc0d95fa76333607c6"} Oct 30 21:32:36.781476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:36.774055 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:32:36.781476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:36.774175 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:32:36.781476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:36.774232 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:32:37.775116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:37.775075 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:32:37.775740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:37.775676 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:32:38.789730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:38.780034 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:32:38.789730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:38.780107 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:32:39.783729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:39.783037 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:32:39.783729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:39.783095 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:32:40.782731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:40.774059 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:32:40.782731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:40.774121 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 21:32:41.664975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.664939 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:41.665229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.665150 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" podUID="146a917a-de8e-4804-a5d0-229ee8c1bead" containerName="maas-api" containerID="cri-o://bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab" gracePeriod=30 Oct 30 21:32:41.778206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.778175 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:32:41.778990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.778972 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:32:41.779597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.779580 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" Oct 30 21:32:41.917455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:41.917389 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:42.053321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053287 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\") pod \"146a917a-de8e-4804-a5d0-229ee8c1bead\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " Oct 30 21:32:42.053489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\" (UniqueName: \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\") pod \"146a917a-de8e-4804-a5d0-229ee8c1bead\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " Oct 30 21:32:42.053489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053472 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\") pod \"146a917a-de8e-4804-a5d0-229ee8c1bead\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " Oct 30 21:32:42.053587 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053548 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\") pod \"146a917a-de8e-4804-a5d0-229ee8c1bead\" (UID: \"146a917a-de8e-4804-a5d0-229ee8c1bead\") " Oct 30 21:32:42.053724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053684 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca" pod "146a917a-de8e-4804-a5d0-229ee8c1bead" (UID: "146a917a-de8e-4804-a5d0-229ee8c1bead"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:42.053889 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.053864 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/146a917a-de8e-4804-a5d0-229ee8c1bead-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.055646 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.055618 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls" pod "146a917a-de8e-4804-a5d0-229ee8c1bead" (UID: "146a917a-de8e-4804-a5d0-229ee8c1bead"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:42.055747 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.055643 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs" pod "146a917a-de8e-4804-a5d0-229ee8c1bead" (UID: "146a917a-de8e-4804-a5d0-229ee8c1bead"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:42.055747 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.055700 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w" pod "146a917a-de8e-4804-a5d0-229ee8c1bead" (UID: "146a917a-de8e-4804-a5d0-229ee8c1bead"). InnerVolumeSpecName "kube-api-access-twp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:42.154904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.154878 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.154904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.154901 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/146a917a-de8e-4804-a5d0-229ee8c1bead-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.155055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.154912 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twp5w\" (UniqueName: \"kubernetes.io/projected/146a917a-de8e-4804-a5d0-229ee8c1bead-kube-api-access-twp5w\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.417687 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.417655 4918 generic.go:356] "Generic (PLEG): container finished" podID="146a917a-de8e-4804-a5d0-229ee8c1bead" containerID="bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab" exitCode=0 Oct 30 21:32:42.417845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.417699 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" Oct 30 21:32:42.417845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.417745 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" event={"ID":"146a917a-de8e-4804-a5d0-229ee8c1bead","Type":"ContainerDied","Data":"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab"} Oct 30 21:32:42.417845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.417802 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z" event={"ID":"146a917a-de8e-4804-a5d0-229ee8c1bead","Type":"ContainerDied","Data":"7e0084293a1da0af91dc1971041c7bfc9939bf430f8923a901f8ac5daafbf739"} Oct 30 21:32:42.417845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.417839 4918 scope.go:122] "RemoveContainer" containerID="bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab" Oct 30 21:32:42.428279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.428256 4918 scope.go:122] "RemoveContainer" containerID="bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab" Oct 30 21:32:42.428616 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:32:42.428591 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab\": container with ID starting with bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab not found: ID does not exist" containerID="bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab" Oct 30 21:32:42.428697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.428627 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab"} err="failed to get container status \"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab\": rpc error: code = NotFound desc = could not find container \"bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab\": container with ID starting with bf86bc7f54af6b1ea5b7a7c41849227bb7260de23b429f7d1c9a1d6b9675e6ab not found: ID does not exist" Oct 30 21:32:42.585578 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.585555 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:42.636319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:42.636297 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2967z"] Oct 30 21:32:43.427215 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:32:43.427182 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146a917a-de8e-4804-a5d0-229ee8c1bead" path="/var/lib/kubelet/pods/146a917a-de8e-4804-a5d0-229ee8c1bead/volumes" Oct 30 21:32:45.800296 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:32:45.800258 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:33:09.471215 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:09.471181 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.668983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:09.668950 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.799780 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:33:15.799741 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:33:30.277100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.277063 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bjd7j"] Oct 30 21:33:30.277481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.277446 4918 cpu_manager.go:414] "containerMap: removing container" podUID="146a917a-de8e-4804-a5d0-229ee8c1bead" containerName="maas-api" Oct 30 21:33:30.277481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.277456 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146a917a-de8e-4804-a5d0-229ee8c1bead" containerName="maas-api" Oct 30 21:33:30.277556 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.277535 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="146a917a-de8e-4804-a5d0-229ee8c1bead" containerName="maas-api" Oct 30 21:33:30.280605 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.280589 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.336904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.336873 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.337086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.336927 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-home\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.337086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.337007 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.337086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.337071 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.337248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.337111 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgv8f\" (UniqueName: \"kubernetes.io/projected/06a6362b-47ea-4e84-8bee-0a50e3643ac3-kube-api-access-dgv8f\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.337248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.337192 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.386034 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.386010 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:33:30.386352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.386331 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6dnp2\"" Oct 30 21:33:30.386454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.386434 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:33:30.386517 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.386469 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:33:30.388966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.388948 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bjd7j"] Oct 30 21:33:30.409774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.409750 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:30.438228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438201 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438239 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-home\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438271 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438302 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438341 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgv8f\" (UniqueName: \"kubernetes.io/projected/06a6362b-47ea-4e84-8bee-0a50e3643ac3-kube-api-access-dgv8f\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438398 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438596 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-home\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438653 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.438770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.438752 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.440584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.440557 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06a6362b-47ea-4e84-8bee-0a50e3643ac3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.440668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.440636 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a6362b-47ea-4e84-8bee-0a50e3643ac3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.501662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.501641 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgv8f\" (UniqueName: \"kubernetes.io/projected/06a6362b-47ea-4e84-8bee-0a50e3643ac3-kube-api-access-dgv8f\") pod \"echo-simulator-kserve-64b6f8f96c-bjd7j\" (UID: \"06a6362b-47ea-4e84-8bee-0a50e3643ac3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.593644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.593585 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:30.967117 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:33:30.967085 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a6362b_47ea_4e84_8bee_0a50e3643ac3.slice/crio-d1588e98053887c0ce640320b2e12241921059641d4eeda83bfddf343e26f322 WatchSource:0}: Error finding container d1588e98053887c0ce640320b2e12241921059641d4eeda83bfddf343e26f322: Status 404 returned error can't find the container with id d1588e98053887c0ce640320b2e12241921059641d4eeda83bfddf343e26f322 Oct 30 21:33:30.970490 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:30.970469 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bjd7j"] Oct 30 21:33:31.685793 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:31.685761 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" event={"ID":"06a6362b-47ea-4e84-8bee-0a50e3643ac3","Type":"ContainerStarted","Data":"d1588e98053887c0ce640320b2e12241921059641d4eeda83bfddf343e26f322"} Oct 30 21:33:33.702732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:33.702674 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" event={"ID":"06a6362b-47ea-4e84-8bee-0a50e3643ac3","Type":"ContainerStarted","Data":"541876df5c578f0d97b1a73145d22d49ed49c4ee884fef1c328c816082e664cb"} Oct 30 21:33:33.823364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:33.823321 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" podStartSLOduration=1.520733248 podStartE2EDuration="3.823308874s" totalImagesPullingTime="2.302575626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:33.822842563 +0000 UTC m=+1194.516324800" watchObservedRunningTime="2026-10-30 21:33:33.823308874 +0000 UTC m=+1194.516791110" Oct 30 21:33:34.320589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.320560 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt"] Oct 30 21:33:34.324088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.324072 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.371814 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.371782 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.371970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.371840 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e5471-ae00-484f-a489-d1670b236686-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.372029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.371964 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9hl\" (UniqueName: \"kubernetes.io/projected/b93e5471-ae00-484f-a489-d1670b236686-kube-api-access-lm9hl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.372029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.372000 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.372138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.372090 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.372195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.372176 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.378134 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.378110 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:33:34.431029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.431003 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt"] Oct 30 21:33:34.473599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473565 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e5471-ae00-484f-a489-d1670b236686-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.473769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473629 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9hl\" (UniqueName: \"kubernetes.io/projected/b93e5471-ae00-484f-a489-d1670b236686-kube-api-access-lm9hl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.473769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473654 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.473769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473685 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.473769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473735 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.473975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.473792 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.474155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.474135 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.474237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.474183 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.474237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.474215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.475840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.475823 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e5471-ae00-484f-a489-d1670b236686-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.476144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.476127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e5471-ae00-484f-a489-d1670b236686-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.519127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.519099 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:34.536314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.536291 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9hl\" (UniqueName: \"kubernetes.io/projected/b93e5471-ae00-484f-a489-d1670b236686-kube-api-access-lm9hl\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nwgzt\" (UID: \"b93e5471-ae00-484f-a489-d1670b236686\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:34.594384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.594337 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:34.637344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:34.637321 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:35.011624 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:33:35.011596 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93e5471_ae00_484f_a489_d1670b236686.slice/crio-64fdb02ff3a1c7f07a2e2e30ce22472c492fdff6f216f99e5ea7dd81fb80c1b6 WatchSource:0}: Error finding container 64fdb02ff3a1c7f07a2e2e30ce22472c492fdff6f216f99e5ea7dd81fb80c1b6: Status 404 returned error can't find the container with id 64fdb02ff3a1c7f07a2e2e30ce22472c492fdff6f216f99e5ea7dd81fb80c1b6 Oct 30 21:33:35.014444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:35.014420 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt"] Oct 30 21:33:35.717074 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:35.717038 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" event={"ID":"b93e5471-ae00-484f-a489-d1670b236686","Type":"ContainerStarted","Data":"64fdb02ff3a1c7f07a2e2e30ce22472c492fdff6f216f99e5ea7dd81fb80c1b6"} Oct 30 21:33:35.991648 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:35.991562 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:37.448523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.448494 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg"] Oct 30 21:33:37.452266 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.452248 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505366 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505340 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505379 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505397 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwrx\" (UniqueName: \"kubernetes.io/projected/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-kube-api-access-2wwrx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505572 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505622 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.505771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.505753 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.556810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.556789 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:33:37.559135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.559116 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg"] Oct 30 21:33:37.580676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.580652 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:37.606976 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.606945 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.606981 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwrx\" (UniqueName: \"kubernetes.io/projected/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-kube-api-access-2wwrx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607038 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607073 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607119 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607184 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607389 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607459 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.607542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.607524 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.610083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.610066 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.610477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.610453 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.671348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.671318 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwrx\" (UniqueName: \"kubernetes.io/projected/95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9-kube-api-access-2wwrx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6gksg\" (UID: \"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.740224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.740152 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" event={"ID":"b93e5471-ae00-484f-a489-d1670b236686","Type":"ContainerStarted","Data":"3251f7c1300712f06d4153b3adfa71af0c8ef441a473c5c00cd3fdafc8d3978c"} Oct 30 21:33:37.766487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.766454 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:37.861702 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:37.861640 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" podStartSLOduration=2.006875995 podStartE2EDuration="3.861619067s" totalImagesPullingTime="1.854743072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:37.859542514 +0000 UTC m=+1198.553024750" watchObservedRunningTime="2026-10-30 21:33:37.861619067 +0000 UTC m=+1198.555101304" Oct 30 21:33:38.146688 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:33:38.146658 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e1109d_e3ce_4f99_a604_7ee8c3bb4fa9.slice/crio-974a0eff9c54f704660d312c448669c9977c8be37b65afa4d7a0df8b37b8dc28 WatchSource:0}: Error finding container 974a0eff9c54f704660d312c448669c9977c8be37b65afa4d7a0df8b37b8dc28: Status 404 returned error can't find the container with id 974a0eff9c54f704660d312c448669c9977c8be37b65afa4d7a0df8b37b8dc28 Oct 30 21:33:38.149318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:38.149230 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg"] Oct 30 21:33:38.637939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:38.637908 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:38.747025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:38.747000 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" event={"ID":"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9","Type":"ContainerStarted","Data":"974a0eff9c54f704660d312c448669c9977c8be37b65afa4d7a0df8b37b8dc28"} Oct 30 21:33:39.538567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.538536 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:33:39.541017 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.540992 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:33:39.551074 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.551057 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:33:39.553493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.553467 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:33:39.556112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.556094 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:33:39.566509 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.566487 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:33:39.753574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.753543 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" event={"ID":"95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9","Type":"ContainerStarted","Data":"be4b54d138f7d4d4da36348e03073f46cfb428fad452f11047bb7667b31b749d"} Oct 30 21:33:39.767058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.767039 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:39.875916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:39.875870 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" podStartSLOduration=2.310904147 podStartE2EDuration="2.875858214s" totalImagesPullingTime="564.954067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:39.873898227 +0000 UTC m=+1200.567380462" watchObservedRunningTime="2026-10-30 21:33:39.875858214 +0000 UTC m=+1200.569340449" Oct 30 21:33:40.594257 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:40.594225 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:40.606753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:40.606692 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:40.680413 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:40.680380 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:40.770644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:40.770616 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bjd7j" Oct 30 21:33:44.606835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:44.606804 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:44.637558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:44.637529 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:44.650903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:44.650881 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:44.798936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:44.798906 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nwgzt" Oct 30 21:33:45.801841 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:33:45.801800 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:33:47.766608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:47.766574 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:47.779568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:47.779548 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:47.808697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:47.808677 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6gksg" Oct 30 21:33:50.139811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:50.139751 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:52.312714 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:52.312666 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:53.523315 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.523282 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt"] Oct 30 21:33:53.527989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.527971 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.632223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.632190 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:33:53.634642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.634622 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt"] Oct 30 21:33:53.645609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.645584 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.645748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.645635 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.645804 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.645771 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.645868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.645827 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4mwv\" (UniqueName: \"kubernetes.io/projected/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-kube-api-access-p4mwv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.645963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.645945 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.646029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.646013 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.651412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.651390 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:53.747142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.747113 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.747292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.747167 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.747463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.747437 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4mwv\" (UniqueName: \"kubernetes.io/projected/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-kube-api-access-p4mwv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.747555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.747538 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.747953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.747923 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.748077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.748030 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.748145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.748080 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.748505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.748476 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.748615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.748523 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.749668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.749646 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.749893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.749879 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.809615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.809559 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4mwv\" (UniqueName: \"kubernetes.io/projected/8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad-kube-api-access-p4mwv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt\" (UID: \"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:53.851283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:53.851258 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:54.229124 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:33:54.229088 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a5e4ee2_bb87_4724_a9ea_b30656c8c6ad.slice/crio-07bab63c4d3630ba5169a2d1f919e5b390d3effd4b0bbe23c6283ec1aae54f89 WatchSource:0}: Error finding container 07bab63c4d3630ba5169a2d1f919e5b390d3effd4b0bbe23c6283ec1aae54f89: Status 404 returned error can't find the container with id 07bab63c4d3630ba5169a2d1f919e5b390d3effd4b0bbe23c6283ec1aae54f89 Oct 30 21:33:54.230970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:54.230947 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt"] Oct 30 21:33:54.841463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:54.841433 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" event={"ID":"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad","Type":"ContainerStarted","Data":"07bab63c4d3630ba5169a2d1f919e5b390d3effd4b0bbe23c6283ec1aae54f89"} Oct 30 21:33:55.854236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:55.854193 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" event={"ID":"8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad","Type":"ContainerStarted","Data":"5c3bd25d58834b6053d41aef48244071c5d020354abf6f6a60e9e780cd75a32a"} Oct 30 21:33:55.982729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:55.982675 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podStartSLOduration=2.372591468 podStartE2EDuration="2.982661336s" totalImagesPullingTime="610.069868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:55.981172626 +0000 UTC m=+1216.674654862" watchObservedRunningTime="2026-10-30 21:33:55.982661336 +0000 UTC m=+1216.676143573" Oct 30 21:33:56.851411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:56.851365 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:33:57.487045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.487011 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7"] Oct 30 21:33:57.490759 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.490735 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.549534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.549508 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7"] Oct 30 21:33:57.580533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580509 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58abe86-5d5d-4384-bab5-23c330ced8a9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.580653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580560 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.580653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580578 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.580787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580762 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhx5\" (UniqueName: \"kubernetes.io/projected/b58abe86-5d5d-4384-bab5-23c330ced8a9-kube-api-access-bzhx5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.580899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580880 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.580942 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.580916 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.596340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.596301 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:33:57.600491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.600469 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:57.681609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681581 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.681767 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681619 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.681767 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681664 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58abe86-5d5d-4384-bab5-23c330ced8a9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.681767 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681692 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.681767 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681736 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.681956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.681774 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhx5\" (UniqueName: \"kubernetes.io/projected/b58abe86-5d5d-4384-bab5-23c330ced8a9-kube-api-access-bzhx5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.682174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.682148 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.682243 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.682214 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.682297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.682279 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.684026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.684005 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58abe86-5d5d-4384-bab5-23c330ced8a9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.684206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.684188 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58abe86-5d5d-4384-bab5-23c330ced8a9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.743431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.743374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhx5\" (UniqueName: \"kubernetes.io/projected/b58abe86-5d5d-4384-bab5-23c330ced8a9-kube-api-access-bzhx5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7\" (UID: \"b58abe86-5d5d-4384-bab5-23c330ced8a9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:57.804657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:57.804631 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:33:58.181151 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:33:58.181107 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58abe86_5d5d_4384_bab5_23c330ced8a9.slice/crio-e324c2d5fe3b1534c7c4582b67194f74fd97780e18dd5077881c41a5656037f7 WatchSource:0}: Error finding container e324c2d5fe3b1534c7c4582b67194f74fd97780e18dd5077881c41a5656037f7: Status 404 returned error can't find the container with id e324c2d5fe3b1534c7c4582b67194f74fd97780e18dd5077881c41a5656037f7 Oct 30 21:33:58.184361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:58.184333 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7"] Oct 30 21:33:58.873201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:58.873158 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" event={"ID":"b58abe86-5d5d-4384-bab5-23c330ced8a9","Type":"ContainerStarted","Data":"48e5d91773ccebfabac50f088ea97365fb6c89ec2ca808671f7944799db200ac"} Oct 30 21:33:58.873596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:58.873223 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" event={"ID":"b58abe86-5d5d-4384-bab5-23c330ced8a9","Type":"ContainerStarted","Data":"e324c2d5fe3b1534c7c4582b67194f74fd97780e18dd5077881c41a5656037f7"} Oct 30 21:33:58.995734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:58.995639 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" podStartSLOduration=1.432639662 podStartE2EDuration="1.995625369s" totalImagesPullingTime="562.985707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:58.992760419 +0000 UTC m=+1219.686242655" watchObservedRunningTime="2026-10-30 21:33:58.995625369 +0000 UTC m=+1219.689107604" Oct 30 21:33:59.805128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:33:59.805098 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:34:03.851653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:03.851621 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:34:03.864103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:03.864080 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:34:03.915144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:03.915117 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:34:07.805186 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:07.805153 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:34:07.817493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:07.817471 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:34:07.939033 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:07.939009 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7" Oct 30 21:34:15.800588 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:34:15.800552 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:34:22.034595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.034564 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:22.041914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.041892 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.053849 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.053815 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:22.091844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.091817 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.091999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.091976 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scck9\" (UniqueName: \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.094469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.094442 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:22.142223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.142198 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:34:22.142326 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.142251 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:34:22.142517 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.142498 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-sszvx\"" Oct 30 21:34:22.193102 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.193079 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.193219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.193153 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scck9\" (UniqueName: \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.193286 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:34:22.193240 4918 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Oct 30 21:34:22.193342 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:34:22.193331 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs podName:f3dca361-23ba-49c7-91c1-e8b9ff8865fd nodeName:}" failed. No retries permitted until 2026-10-30 21:34:22.693301534 +0000 UTC m=+1243.386783766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs") pod "ai-gateway-controller-6d49f5bf5d-wjkzz" (UID: "f3dca361-23ba-49c7-91c1-e8b9ff8865fd") : secret "ai-gateway-controller-metrics-tls" not found Oct 30 21:34:22.255288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.255259 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scck9\" (UniqueName: \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.381212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.381175 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:22.698595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.698525 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.700894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.700869 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-wjkzz\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:22.958628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:22.958546 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:23.330355 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:34:23.330327 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dca361_23ba_49c7_91c1_e8b9ff8865fd.slice/crio-12dc53a7160babae95f81f92390acef4457aab139de488e5c2784a32616fc2c3 WatchSource:0}: Error finding container 12dc53a7160babae95f81f92390acef4457aab139de488e5c2784a32616fc2c3: Status 404 returned error can't find the container with id 12dc53a7160babae95f81f92390acef4457aab139de488e5c2784a32616fc2c3 Oct 30 21:34:23.333029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:23.332961 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:24.023561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:24.023527 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" event={"ID":"f3dca361-23ba-49c7-91c1-e8b9ff8865fd","Type":"ContainerStarted","Data":"12dc53a7160babae95f81f92390acef4457aab139de488e5c2784a32616fc2c3"} Oct 30 21:34:26.037405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:26.037370 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" event={"ID":"f3dca361-23ba-49c7-91c1-e8b9ff8865fd","Type":"ContainerStarted","Data":"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574"} Oct 30 21:34:26.037837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:26.037430 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:26.162611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:26.162551 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" podStartSLOduration=2.886544496 podStartE2EDuration="5.162533741s" totalImagesPullingTime="2.275989245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:26.158031225 +0000 UTC m=+1246.851513461" watchObservedRunningTime="2026-10-30 21:34:26.162533741 +0000 UTC m=+1246.856015978" Oct 30 21:34:27.657654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.657621 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:34:27.665852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.665829 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.746739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.746690 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.746889 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.746744 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.746889 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.746831 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.746963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.746889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.746963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.746949 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.747030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.747020 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.747063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.747048 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmsv\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.747096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.747079 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.747150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.747138 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.774899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.774869 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:34:27.848251 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848220 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848258 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848380 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmsv\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848433 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848469 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848523 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.848931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848903 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.848927 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849017 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849068 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849159 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849227 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849286 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849250 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.849583 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.849564 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.850667 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.850633 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.850888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.850871 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.910741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.910665 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.910741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.910668 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmsv\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\") pod \"maas-default-gateway-openshift-default-67bf986f99-kd64s\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:27.981391 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:27.981367 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:28.398955 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:34:28.398919 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03581726_86b6_4ad3_b334_8161899bf209.slice/crio-6af5055c800a18a4a4490e4dec32a2de7523952a10f277d87bc58169a2af9b3a WatchSource:0}: Error finding container 6af5055c800a18a4a4490e4dec32a2de7523952a10f277d87bc58169a2af9b3a: Status 404 returned error can't find the container with id 6af5055c800a18a4a4490e4dec32a2de7523952a10f277d87bc58169a2af9b3a Oct 30 21:34:28.401230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:28.401197 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:28.401309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:28.401265 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:28.401349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:28.401313 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:28.401796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:28.401759 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:34:29.058551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.058518 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" event={"ID":"03581726-86b6-4ad3-b334-8161899bf209","Type":"ContainerStarted","Data":"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9"} Oct 30 21:34:29.058972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.058573 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" event={"ID":"03581726-86b6-4ad3-b334-8161899bf209","Type":"ContainerStarted","Data":"6af5055c800a18a4a4490e4dec32a2de7523952a10f277d87bc58169a2af9b3a"} Oct 30 21:34:29.294729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.294425 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podStartSLOduration=2.294395128 podStartE2EDuration="2.294395128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:29.233802631 +0000 UTC m=+1249.927284867" watchObservedRunningTime="2026-10-30 21:34:29.294395128 +0000 UTC m=+1249.987877364" Oct 30 21:34:29.600302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.600263 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.799674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.799641 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.995872 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.995836 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:29.997019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.996944 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xplfs"] Oct 30 21:34:29.997473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.997437 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:29.997661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:29.997627 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:30.001542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.001523 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.033780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.033757 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rc98d-vb8mz"] Oct 30 21:34:30.042436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.042409 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.048905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.048882 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xplfs"] Oct 30 21:34:30.055055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.055031 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7ntcx\"" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092355 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092429 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d73838-6d45-480e-8964-eba9637c8a09-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092513 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5fmd\" (UniqueName: \"kubernetes.io/projected/b8d73838-6d45-480e-8964-eba9637c8a09-kube-api-access-b5fmd\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092541 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092611 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t5s\" (UniqueName: \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092640 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.107612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.092843 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:34:30.109505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.109474 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rc98d-vb8mz"] Oct 30 21:34:30.136948 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.136926 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gnxv2"] Oct 30 21:34:30.140780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.140757 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.155247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.155228 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nvvjf\"" Oct 30 21:34:30.194242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.194208 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5fmd\" (UniqueName: \"kubernetes.io/projected/b8d73838-6d45-480e-8964-eba9637c8a09-kube-api-access-b5fmd\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.194503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.194479 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.194722 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.194684 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t5s\" (UniqueName: \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.197157 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.197137 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.197844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.197823 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41233f0-deeb-4c61-b54f-e5334219b9a5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.198094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.196674 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.198183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.198069 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.198450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.198374 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qnz\" (UniqueName: \"kubernetes.io/projected/b41233f0-deeb-4c61-b54f-e5334219b9a5-kube-api-access-p5qnz\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.198537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.198521 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d73838-6d45-480e-8964-eba9637c8a09-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.198771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.198747 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.200068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.200047 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d73838-6d45-480e-8964-eba9637c8a09-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.201128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.201104 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.204604 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.204583 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gnxv2"] Oct 30 21:34:30.257884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.257826 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5fmd\" (UniqueName: \"kubernetes.io/projected/b8d73838-6d45-480e-8964-eba9637c8a09-kube-api-access-b5fmd\") pod \"payload-pre-processing-7b98bb678c-xplfs\" (UID: \"b8d73838-6d45-480e-8964-eba9637c8a09\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.258322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.258304 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t5s\" (UniqueName: \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\") pod \"periodic-gathering-rc98d-vb8mz\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.300452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.300422 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41233f0-deeb-4c61-b54f-e5334219b9a5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.300690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.300663 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qnz\" (UniqueName: \"kubernetes.io/projected/b41233f0-deeb-4c61-b54f-e5334219b9a5-kube-api-access-p5qnz\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.301248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.301230 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41233f0-deeb-4c61-b54f-e5334219b9a5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.315114 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.315088 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:30.360553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.360525 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:34:30.401888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.398395 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qnz\" (UniqueName: \"kubernetes.io/projected/b41233f0-deeb-4c61-b54f-e5334219b9a5-kube-api-access-p5qnz\") pod \"payload-processing-5f78687d5b-gnxv2\" (UID: \"b41233f0-deeb-4c61-b54f-e5334219b9a5\") " pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.454700 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.454670 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:30.720849 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:34:30.720814 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d73838_6d45_480e_8964_eba9637c8a09.slice/crio-df9306285893e165839f070414300c28d089bd5bf71ba26ec1c2a80724a0cd74 WatchSource:0}: Error finding container df9306285893e165839f070414300c28d089bd5bf71ba26ec1c2a80724a0cd74: Status 404 returned error can't find the container with id df9306285893e165839f070414300c28d089bd5bf71ba26ec1c2a80724a0cd74 Oct 30 21:34:30.724513 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.724491 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xplfs"] Oct 30 21:34:30.768935 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:34:30.768903 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20f97ea_11d1_439a_b9ad_4a2218f64be7.slice/crio-0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab WatchSource:0}: Error finding container 0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab: Status 404 returned error can't find the container with id 0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab Oct 30 21:34:30.802630 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.794929 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rc98d-vb8mz"] Oct 30 21:34:30.873736 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:34:30.869771 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41233f0_deeb_4c61_b54f_e5334219b9a5.slice/crio-116436c6571fe39061d60f810b62ebef6c2520ca8b482dc69119463d6d78c801 WatchSource:0}: Error finding container 116436c6571fe39061d60f810b62ebef6c2520ca8b482dc69119463d6d78c801: Status 404 returned error can't find the container with id 116436c6571fe39061d60f810b62ebef6c2520ca8b482dc69119463d6d78c801 Oct 30 21:34:30.897456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.896884 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gnxv2"] Oct 30 21:34:30.989854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.989795 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:30.990060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:30.989866 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:31.101071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.100979 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" event={"ID":"c20f97ea-11d1-439a-b9ad-4a2218f64be7","Type":"ContainerStarted","Data":"3e79533a6262db056d428579747c46fa8a290fb7a54a522cb94819fb2d033cf3"} Oct 30 21:34:31.101071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.101032 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" event={"ID":"c20f97ea-11d1-439a-b9ad-4a2218f64be7","Type":"ContainerStarted","Data":"0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab"} Oct 30 21:34:31.102942 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.102914 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" event={"ID":"b41233f0-deeb-4c61-b54f-e5334219b9a5","Type":"ContainerStarted","Data":"116436c6571fe39061d60f810b62ebef6c2520ca8b482dc69119463d6d78c801"} Oct 30 21:34:31.104258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.104232 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" event={"ID":"b8d73838-6d45-480e-8964-eba9637c8a09","Type":"ContainerStarted","Data":"df9306285893e165839f070414300c28d089bd5bf71ba26ec1c2a80724a0cd74"} Oct 30 21:34:31.226608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.226539 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" podStartSLOduration=2.226524113 podStartE2EDuration="2.226524113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:31.224768905 +0000 UTC m=+1251.918251141" watchObservedRunningTime="2026-10-30 21:34:31.226524113 +0000 UTC m=+1251.920006345" Oct 30 21:34:31.993501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.993061 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:31.993501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:31.993121 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:32.996145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:32.996103 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:32.996645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:32.996164 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:33.984730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:33.982068 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:33.984730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:33.982130 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:34.981674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:34.981592 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-kd64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:34:34.981674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:34.981644 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 21:34:35.137945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.137911 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" event={"ID":"b41233f0-deeb-4c61-b54f-e5334219b9a5","Type":"ContainerStarted","Data":"4dc361b5933840fd5d5403c60a17c4c9c4f800e0ed809d85b5309b4e98ed71a1"} Oct 30 21:34:35.138143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.138002 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:35.139584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.139555 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" event={"ID":"b8d73838-6d45-480e-8964-eba9637c8a09","Type":"ContainerStarted","Data":"998cd86777cd93bdf86a8a49785b9623022b68c65970f9beebeb49111cd581c8"} Oct 30 21:34:35.139741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.139641 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:35.260122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.260024 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" podStartSLOduration=1.517326298 podStartE2EDuration="5.260007661s" totalImagesPullingTime="3.742681363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:35.259474153 +0000 UTC m=+1255.952956426" watchObservedRunningTime="2026-10-30 21:34:35.260007661 +0000 UTC m=+1255.953489899" Oct 30 21:34:35.378154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:35.378102 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" podStartSLOduration=2.449453798 podStartE2EDuration="6.378088964s" totalImagesPullingTime="3.928635166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:35.377054463 +0000 UTC m=+1256.070536699" watchObservedRunningTime="2026-10-30 21:34:35.378088964 +0000 UTC m=+1256.071571199" Oct 30 21:34:36.390974 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:36.390938 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:36.794097 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:36.794017 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:36.794460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:36.794433 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:36.795124 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:36.795105 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:34:37.045598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:37.045519 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:40.822625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:40.822585 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:45.801623 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:34:45.801585 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:34:46.150266 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:46.150239 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xplfs" Oct 30 21:34:46.150417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:46.150315 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-gnxv2" Oct 30 21:34:52.268823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.268790 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:52.269290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.269048 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" podUID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" containerName="manager" containerID="cri-o://8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574" gracePeriod=30 Oct 30 21:34:52.525216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.525164 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:52.614675 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.614648 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") pod \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " Oct 30 21:34:52.614796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.614729 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\" (UniqueName: \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\") pod \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\" (UID: \"f3dca361-23ba-49c7-91c1-e8b9ff8865fd\") " Oct 30 21:34:52.616800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.616775 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs" pod "f3dca361-23ba-49c7-91c1-e8b9ff8865fd" (UID: "f3dca361-23ba-49c7-91c1-e8b9ff8865fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:52.616800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.616795 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9" pod "f3dca361-23ba-49c7-91c1-e8b9ff8865fd" (UID: "f3dca361-23ba-49c7-91c1-e8b9ff8865fd"). InnerVolumeSpecName "kube-api-access-scck9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:52.716269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.716240 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:52.716269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:52.716266 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scck9\" (UniqueName: \"kubernetes.io/projected/f3dca361-23ba-49c7-91c1-e8b9ff8865fd-kube-api-access-scck9\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:53.250146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.250110 4918 generic.go:356] "Generic (PLEG): container finished" podID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" containerID="8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574" exitCode=0 Oct 30 21:34:53.250314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.250173 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" event={"ID":"f3dca361-23ba-49c7-91c1-e8b9ff8865fd","Type":"ContainerDied","Data":"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574"} Oct 30 21:34:53.250314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.250185 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" Oct 30 21:34:53.250314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.250214 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz" event={"ID":"f3dca361-23ba-49c7-91c1-e8b9ff8865fd","Type":"ContainerDied","Data":"12dc53a7160babae95f81f92390acef4457aab139de488e5c2784a32616fc2c3"} Oct 30 21:34:53.250314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.250233 4918 scope.go:122] "RemoveContainer" containerID="8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574" Oct 30 21:34:53.263476 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.263453 4918 scope.go:122] "RemoveContainer" containerID="8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574" Oct 30 21:34:53.263903 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:34:53.263885 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574\": container with ID starting with 8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574 not found: ID does not exist" containerID="8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574" Oct 30 21:34:53.263955 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.263911 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574"} err="failed to get container status \"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574\": rpc error: code = NotFound desc = could not find container \"8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574\": container with ID starting with 8224bcd0f54c179a33f5a9939296abdad29754dd4ca4b7244fd041681e7ce574 not found: ID does not exist" Oct 30 21:34:53.418195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.418165 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:53.468679 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.468654 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-wjkzz"] Oct 30 21:34:53.959645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:53.959615 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:54.521622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:54.521586 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.426979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:55.426947 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" path="/var/lib/kubelet/pods/f3dca361-23ba-49c7-91c1-e8b9ff8865fd/volumes" Oct 30 21:34:56.311150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:56.311119 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:56.501647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:56.501612 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:57.980317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:57.980282 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:58.227156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:34:58.227121 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:15.801557 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:15.801520 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:35:26.775066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:26.774978 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:26.970365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:26.970329 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:27.459467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.459428 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:27.614675 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.614645 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:35:27.615132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.615119 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" containerName="manager" Oct 30 21:35:27.615176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.615133 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" containerName="manager" Oct 30 21:35:27.615229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.615219 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3dca361-23ba-49c7-91c1-e8b9ff8865fd" containerName="manager" Oct 30 21:35:27.618427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.618406 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646333 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646371 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646397 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv7k\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646600 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646694 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646762 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646792 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646922 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.646977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.646937 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.662284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.662256 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:27.675122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.675017 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ed76b8-openshift-default-dockercfg-47s56\"" Oct 30 21:35:27.727593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.727540 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:35:27.748055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748025 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748069 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748110 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748141 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748167 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv7k\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748231 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748286 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748461 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748461 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748339 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748674 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748801 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748819 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.748948 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748919 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.749006 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.748995 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.750325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.750305 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.750693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.750675 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.810700 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.810667 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.811118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.811097 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv7k\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:27.855426 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.855392 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:27.931790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:27.931764 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:28.041618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.041565 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.239887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.239855 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 21:35:28.245751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.245604 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.251020 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.250992 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.318042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.318005 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 21:35:28.323138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.323108 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-19dd4f-openshift-default-dockercfg-rbrph\"" Oct 30 21:35:28.355027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355000 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355043 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355157 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355191 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355218 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355315 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355264 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrml\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355351 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355398 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.355475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.355459 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.377435 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:28.377388 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e09117f_7e66_4e32_9b3d_5e4058fde4fc.slice/crio-379aac9bf42004f43c1bd56503387486b8e8f9b2d154781298b5d492e618d443 WatchSource:0}: Error finding container 379aac9bf42004f43c1bd56503387486b8e8f9b2d154781298b5d492e618d443: Status 404 returned error can't find the container with id 379aac9bf42004f43c1bd56503387486b8e8f9b2d154781298b5d492e618d443 Oct 30 21:35:28.379702 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.379666 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:28.379835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.379755 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:28.379835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.379793 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:28.383807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.383784 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:35:28.456819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456791 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.456956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456835 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.456956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456877 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.456956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456910 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.456956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456946 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.456970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrml\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457009 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457077 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457111 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457361 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457457 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457670 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457678 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.457945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.457776 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.459394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.459374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.459618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.459599 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.471522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.471493 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerStarted","Data":"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b"} Oct 30 21:35:28.471660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.471543 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerStarted","Data":"379aac9bf42004f43c1bd56503387486b8e8f9b2d154781298b5d492e618d443"} Oct 30 21:35:28.497530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.497457 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.524770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.524736 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.525080 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.525058 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrml\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.576164 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.576133 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:28.601262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.601215 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podStartSLOduration=1.6011993389999999 podStartE2EDuration="1.601199339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:28.599197034 +0000 UTC m=+1309.292679281" watchObservedRunningTime="2026-10-30 21:35:28.601199339 +0000 UTC m=+1309.294681647" Oct 30 21:35:28.689084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.689044 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.871264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.871224 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.900562 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.900532 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:35:28.905477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.905455 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.932350 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.932309 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:28.960011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.959980 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71aab5-openshift-default-dockercfg-hxtfl\"" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.962907 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.962963 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963052 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963091 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963142 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963179 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj4sp\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963236 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963273 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.963491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.963314 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:28.966795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.966427 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:35:28.970608 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:28.970569 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5a2312_fa87_4a37_9202_2ab94864be0f.slice/crio-99693ad7fbdfe089f6324741cb8897913d886232584e9aa588188dbc75e0aab9 WatchSource:0}: Error finding container 99693ad7fbdfe089f6324741cb8897913d886232584e9aa588188dbc75e0aab9: Status 404 returned error can't find the container with id 99693ad7fbdfe089f6324741cb8897913d886232584e9aa588188dbc75e0aab9 Oct 30 21:35:28.970971 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.970953 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 21:35:28.973646 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.973610 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:28.973762 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.973691 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:28.973852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:28.973802 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:29.061530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.061498 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.064369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064344 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064390 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064431 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj4sp\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064516 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064556 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064787 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064845 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064925 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064902 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.064987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064944 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.065039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.064983 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.065160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.065138 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.065226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.065145 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.065512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.065487 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.066941 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.066924 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.067040 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.066948 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.126323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.126255 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj4sp\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.129330 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.129303 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.220825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.220794 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:29.333590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.333439 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.479242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.479146 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerStarted","Data":"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2"} Oct 30 21:35:29.479242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.479211 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerStarted","Data":"99693ad7fbdfe089f6324741cb8897913d886232584e9aa588188dbc75e0aab9"} Oct 30 21:35:29.528548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.528520 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.576549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.576513 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:29.606043 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.605987 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podStartSLOduration=1.6059727769999999 podStartE2EDuration="1.605972777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:29.602642098 +0000 UTC m=+1310.296124333" watchObservedRunningTime="2026-10-30 21:35:29.605972777 +0000 UTC m=+1310.299455012" Oct 30 21:35:29.614994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.614968 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:35:29.644163 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:29.644139 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd625a7ac_414e_43ee_a38e_d6a0b65ed6be.slice/crio-06424e29c1b2f04ecc3352e1ffdab98d836599a43da993b7b84aaf1411aba900 WatchSource:0}: Error finding container 06424e29c1b2f04ecc3352e1ffdab98d836599a43da993b7b84aaf1411aba900: Status 404 returned error can't find the container with id 06424e29c1b2f04ecc3352e1ffdab98d836599a43da993b7b84aaf1411aba900 Oct 30 21:35:29.646464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.646428 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:29.646557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.646535 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:29.646597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.646581 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:29.725066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.725034 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.937898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:29.937867 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:30.486574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.486542 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerStarted","Data":"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75"} Oct 30 21:35:30.486744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.486592 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerStarted","Data":"06424e29c1b2f04ecc3352e1ffdab98d836599a43da993b7b84aaf1411aba900"} Oct 30 21:35:30.487050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.487031 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:30.487900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.487881 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:35:30.582341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.582306 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:30.611773 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:30.611722 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podStartSLOduration=2.611696677 podStartE2EDuration="2.611696677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:30.609206255 +0000 UTC m=+1311.302688491" watchObservedRunningTime="2026-10-30 21:35:30.611696677 +0000 UTC m=+1311.305178945" Oct 30 21:35:31.221386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.221345 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:31.226718 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.226679 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:31.493431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.493293 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:31.493431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.493347 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:31.494189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.494167 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:35:31.494317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.494220 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:35:31.522080 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.522045 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:31.717253 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:31.717216 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:32.186524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:32.186486 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:32.371676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:32.371645 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:33.132408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:33.132372 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:33.353099 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:33.353057 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:33.558490 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:33.558401 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:33.924958 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:33.924923 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:34.125502 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:34.125471 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.154506 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.154461 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.346193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.346157 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.530255 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.530160 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 21:35:36.536574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.536542 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.568826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.568797 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.636977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.636942 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 21:35:36.645753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.644420 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ed76b8\"" Oct 30 21:35:36.645753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.644436 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ed76b8-dockercfg-kbllq\"" Oct 30 21:35:36.649949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.649902 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.650089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.650051 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8hj\" (UniqueName: \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.698463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.698432 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 21:35:36.702455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.702434 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.741020 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.740988 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 21:35:36.751008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.750980 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmsmd\" (UniqueName: \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.751142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.751103 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8hj\" (UniqueName: \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.751188 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.751155 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.751273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.751257 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.752031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.752013 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.762079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.762047 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.813569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.813498 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8hj\" (UniqueName: \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\") pod \"payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.851128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.851084 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:36.852446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.852419 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.852565 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.852541 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmsmd\" (UniqueName: \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.853188 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.853169 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:36.923762 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:36.922992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmsmd\" (UniqueName: \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\") pod \"payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:37.016634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.016595 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:37.177868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.177830 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.229497 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:37.229450 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d081d8_3123_4d0c_b73c_fa690d3b1156.slice/crio-3e72b506509e3a95512eb8ddc2d3bd98d3df1c75495df90de70f333baaad7582 WatchSource:0}: Error finding container 3e72b506509e3a95512eb8ddc2d3bd98d3df1c75495df90de70f333baaad7582: Status 404 returned error can't find the container with id 3e72b506509e3a95512eb8ddc2d3bd98d3df1c75495df90de70f333baaad7582 Oct 30 21:35:37.230320 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.230204 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 21:35:37.375630 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.375585 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.387639 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:37.387613 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc1cf7c_1aa4_4cca_92bf_7aae2b33a100.slice/crio-2b9b4910054034b89459ba313e65448e6dda77e62d0fd65920844fa04a3e2e17 WatchSource:0}: Error finding container 2b9b4910054034b89459ba313e65448e6dda77e62d0fd65920844fa04a3e2e17: Status 404 returned error can't find the container with id 2b9b4910054034b89459ba313e65448e6dda77e62d0fd65920844fa04a3e2e17 Oct 30 21:35:37.391306 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.391283 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 21:35:37.470837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.470812 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 21:35:37.474647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.474626 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.532391 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.532356 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 21:35:37.535288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.535252 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" event={"ID":"68d081d8-3123-4d0c-b73c-fa690d3b1156","Type":"ContainerStarted","Data":"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c"} Oct 30 21:35:37.535464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.535313 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" event={"ID":"68d081d8-3123-4d0c-b73c-fa690d3b1156","Type":"ContainerStarted","Data":"3e72b506509e3a95512eb8ddc2d3bd98d3df1c75495df90de70f333baaad7582"} Oct 30 21:35:37.535464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.535401 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:37.537596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.537567 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" event={"ID":"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100","Type":"ContainerStarted","Data":"8cc74bc6dd971b9cc28d3636dd045482a5e41cb454a98899d5ed10110bfa71c4"} Oct 30 21:35:37.537736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.537618 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" event={"ID":"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100","Type":"ContainerStarted","Data":"2b9b4910054034b89459ba313e65448e6dda77e62d0fd65920844fa04a3e2e17"} Oct 30 21:35:37.537852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.537834 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:37.560272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.560234 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.560422 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.560287 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.560467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.560420 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.560533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.560517 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xb5\" (UniqueName: \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.579036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.579007 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-554f46-serving-cert\"" Oct 30 21:35:37.579144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.579072 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hkklw\"" Oct 30 21:35:37.579144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.579082 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:35:37.661769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.661666 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.661947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.661785 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xb5\" (UniqueName: \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.661947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.661898 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.662061 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.661947 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.662122 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:37.662073 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-serving-cert: secret "maas-api-e2e-worker-w4-554f46-serving-cert" not found Oct 30 21:35:37.662176 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:37.662133 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls podName:db507623-9d17-4e5e-b1b9-d1d9084b106b nodeName:}" failed. No retries permitted until 2026-10-30 21:35:38.162117489 +0000 UTC m=+1318.855599731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls") pod "maas-api-e2e-worker-w4-554f46-867d78d79-98njf" (UID: "db507623-9d17-4e5e-b1b9-d1d9084b106b") : secret "maas-api-e2e-worker-w4-554f46-serving-cert" not found Oct 30 21:35:37.662549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.662527 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.664362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.664344 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.707223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.707192 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.728783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.728738 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" podStartSLOduration=1.728722885 podStartE2EDuration="1.728722885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:37.728070056 +0000 UTC m=+1318.421552295" watchObservedRunningTime="2026-10-30 21:35:37.728722885 +0000 UTC m=+1318.422205123" Oct 30 21:35:37.738299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.738271 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xb5\" (UniqueName: \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:37.869932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.869874 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" podStartSLOduration=1.869856403 podStartE2EDuration="1.869856403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:37.868999494 +0000 UTC m=+1318.562481730" watchObservedRunningTime="2026-10-30 21:35:37.869856403 +0000 UTC m=+1318.563338641" Oct 30 21:35:37.927685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:37.927596 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:38.166814 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:38.166775 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:38.169309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:38.169285 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-554f46-867d78d79-98njf\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:38.388493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:38.388458 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:38.809569 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:38.809523 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb507623_9d17_4e5e_b1b9_d1d9084b106b.slice/crio-7f2f819422936bf6115efbd9c85d4e57a78eb92182e52bf1c238cadab9634b7e WatchSource:0}: Error finding container 7f2f819422936bf6115efbd9c85d4e57a78eb92182e52bf1c238cadab9634b7e: Status 404 returned error can't find the container with id 7f2f819422936bf6115efbd9c85d4e57a78eb92182e52bf1c238cadab9634b7e Oct 30 21:35:38.811434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:38.811412 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 21:35:38.838239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:38.838208 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:39.024920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:39.024886 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:39.551379 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:39.551340 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" event={"ID":"db507623-9d17-4e5e-b1b9-d1d9084b106b","Type":"ContainerStarted","Data":"7f2f819422936bf6115efbd9c85d4e57a78eb92182e52bf1c238cadab9634b7e"} Oct 30 21:35:40.559874 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:40.559838 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" event={"ID":"db507623-9d17-4e5e-b1b9-d1d9084b106b","Type":"ContainerStarted","Data":"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553"} Oct 30 21:35:40.560298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:40.559928 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:40.684849 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:40.684796 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" podStartSLOduration=2.785814933 podStartE2EDuration="3.684783118s" totalImagesPullingTime="898.968185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:40.6833723 +0000 UTC m=+1321.376854533" watchObservedRunningTime="2026-10-30 21:35:40.684783118 +0000 UTC m=+1321.378265354" Oct 30 21:35:41.156695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:41.156657 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:41.346537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:41.346499 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:42.464535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.464485 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 21:35:42.468945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.468919 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.513666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.513630 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4g5\" (UniqueName: \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.513829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.513694 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.513894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.513873 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.513957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.513909 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.527736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.527678 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 21:35:42.574929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.574898 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a2dced-serving-cert\"" Oct 30 21:35:42.615142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.615111 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4g5\" (UniqueName: \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.615355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.615171 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.615355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.615271 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.615355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.615316 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.615518 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:42.615404 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-serving-cert: secret "maas-api-e2e-worker-w6-a2dced-serving-cert" not found Oct 30 21:35:42.615518 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:42.615476 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls podName:26b7f92c-07d9-4272-a240-9966ac95e81f nodeName:}" failed. No retries permitted until 2026-10-30 21:35:43.115454464 +0000 UTC m=+1323.808936694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls") pod "maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" (UID: "26b7f92c-07d9-4272-a240-9966ac95e81f") : secret "maas-api-e2e-worker-w6-a2dced-serving-cert" not found Oct 30 21:35:42.616025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.616001 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.617752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.617730 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.679534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.679493 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4g5\" (UniqueName: \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:42.695151 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.695118 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:42.885469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:42.885436 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.120518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.120480 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:43.123157 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.123129 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:43.383451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.383413 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:43.736612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.736509 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.762798 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:43.762761 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b7f92c_07d9_4272_a240_9966ac95e81f.slice/crio-b6e960ff331128eeae39fbd1d9ef84502bd6f8670ace7d9de45884a181796763 WatchSource:0}: Error finding container b6e960ff331128eeae39fbd1d9ef84502bd6f8670ace7d9de45884a181796763: Status 404 returned error can't find the container with id b6e960ff331128eeae39fbd1d9ef84502bd6f8670ace7d9de45884a181796763 Oct 30 21:35:43.770481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.767519 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 21:35:43.948137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:43.948092 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.586004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:44.585961 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" event={"ID":"26b7f92c-07d9-4272-a240-9966ac95e81f","Type":"ContainerStarted","Data":"b6e960ff331128eeae39fbd1d9ef84502bd6f8670ace7d9de45884a181796763"} Oct 30 21:35:45.594960 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:45.594919 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" event={"ID":"26b7f92c-07d9-4272-a240-9966ac95e81f","Type":"ContainerStarted","Data":"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588"} Oct 30 21:35:45.595378 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:45.595128 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:45.717234 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:45.717171 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" podStartSLOduration=2.789152192 podStartE2EDuration="3.717151392s" totalImagesPullingTime="927.9992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:45.714923637 +0000 UTC m=+1326.408405873" watchObservedRunningTime="2026-10-30 21:35:45.717151392 +0000 UTC m=+1326.410633630" Oct 30 21:35:45.806244 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:35:45.806203 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:35:46.303423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:46.303391 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:46.490091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:46.490054 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:46.578331 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:46.578263 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 21:35:47.859786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:47.859746 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:48.057280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:48.057239 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:48.245093 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:48.245001 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:48.547126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:48.547042 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 21:35:48.547126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:48.547105 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 21:35:48.935046 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:48.935012 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:49.141109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.141071 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:49.356868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.356788 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 21:35:49.360899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.360881 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.451233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.451194 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 21:35:49.462558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.462533 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a2dced\"" Oct 30 21:35:49.462558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.462538 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a2dced-dockercfg-nnxzr\"" Oct 30 21:35:49.532504 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.532457 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.532666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.532599 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn56m\" (UniqueName: \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.633411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.633376 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.633603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.633477 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn56m\" (UniqueName: \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.634174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.634155 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.696242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.696215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn56m\" (UniqueName: \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\") pod \"payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:49.975102 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:49.975017 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:50.347891 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:50.347854 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164c817d_3e33_4d6f_b000_6ea2d1f487a8.slice/crio-b617e2323d17ab0a4e72ace879c62766478acd3de611fd6438993cdeb38f2c67 WatchSource:0}: Error finding container b617e2323d17ab0a4e72ace879c62766478acd3de611fd6438993cdeb38f2c67: Status 404 returned error can't find the container with id b617e2323d17ab0a4e72ace879c62766478acd3de611fd6438993cdeb38f2c67 Oct 30 21:35:50.350902 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:50.350874 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 21:35:50.627199 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:50.627162 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" event={"ID":"164c817d-3e33-4d6f-b000-6ea2d1f487a8","Type":"ContainerStarted","Data":"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3"} Oct 30 21:35:50.627377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:50.627222 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" event={"ID":"164c817d-3e33-4d6f-b000-6ea2d1f487a8","Type":"ContainerStarted","Data":"b617e2323d17ab0a4e72ace879c62766478acd3de611fd6438993cdeb38f2c67"} Oct 30 21:35:50.627449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:50.627417 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:35:50.750534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:50.750468 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" podStartSLOduration=1.750452692 podStartE2EDuration="1.750452692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:50.749613352 +0000 UTC m=+1331.443095588" watchObservedRunningTime="2026-10-30 21:35:50.750452692 +0000 UTC m=+1331.443934928" Oct 30 21:35:51.571529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:51.571497 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:51.614139 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:51.614108 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 21:35:51.770271 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:51.770225 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:52.086374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.086333 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 21:35:52.091457 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.091431 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.138037 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.137999 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 21:35:52.145270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.145232 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-991335\"" Oct 30 21:35:52.145479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.145459 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-991335-dockercfg-zf9fj\"" Oct 30 21:35:52.271987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.271918 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctt2v\" (UniqueName: \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.272155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.272072 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.373970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.373927 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctt2v\" (UniqueName: \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.374158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.373999 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.374941 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.374911 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.436047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.436022 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctt2v\" (UniqueName: \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\") pod \"payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:52.709374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:52.709290 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:53.059666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.059521 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:53.084601 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:35:53.084567 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ec3b7_9a1c_4e86_92f6_4a2f6df0274b.slice/crio-c27e234ca9b3f2bb44b32dc9e358dc8967d329945bbf96031b2d95c4cb07b0c5 WatchSource:0}: Error finding container c27e234ca9b3f2bb44b32dc9e358dc8967d329945bbf96031b2d95c4cb07b0c5: Status 404 returned error can't find the container with id c27e234ca9b3f2bb44b32dc9e358dc8967d329945bbf96031b2d95c4cb07b0c5 Oct 30 21:35:53.087454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.087426 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 21:35:53.649963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.649928 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" event={"ID":"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b","Type":"ContainerStarted","Data":"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318"} Oct 30 21:35:53.650141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.649977 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" event={"ID":"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b","Type":"ContainerStarted","Data":"c27e234ca9b3f2bb44b32dc9e358dc8967d329945bbf96031b2d95c4cb07b0c5"} Oct 30 21:35:53.650141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.650034 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:35:53.770138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:53.770073 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" podStartSLOduration=1.770060445 podStartE2EDuration="1.770060445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:53.767938263 +0000 UTC m=+1334.461420499" watchObservedRunningTime="2026-10-30 21:35:53.770060445 +0000 UTC m=+1334.463542682" Oct 30 21:35:56.621979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:56.621941 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:56.803506 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:56.803456 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:58.160059 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:58.160012 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:58.363242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:58.363200 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:58.556868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:35:58.556772 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:01.640187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:01.640090 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 21:36:02.534954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:02.534897 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:36:02.535135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:02.534984 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:03.147405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:03.147366 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:04.662282 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.662252 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 21:36:04.752509 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.752420 4918 generic.go:356] "Generic (PLEG): container finished" podID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerID="a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2" exitCode=0 Oct 30 21:36:04.752674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.752499 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerDied","Data":"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2"} Oct 30 21:36:04.752963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.752947 4918 scope.go:122] "RemoveContainer" containerID="a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2" Oct 30 21:36:04.754425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.754398 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:04.754491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.754466 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:04.754526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:04.754496 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:05.760691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:05.760654 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerStarted","Data":"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70"} Oct 30 21:36:06.585110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:06.585075 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:36:06.586589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:06.586559 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:36:06.586806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:06.586780 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:36:06.774331 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:06.774294 4918 generic.go:356] "Generic (PLEG): container finished" podID="c20f97ea-11d1-439a-b9ad-4a2218f64be7" containerID="3e79533a6262db056d428579747c46fa8a290fb7a54a522cb94819fb2d033cf3" exitCode=0 Oct 30 21:36:06.775752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:06.775725 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" event={"ID":"c20f97ea-11d1-439a-b9ad-4a2218f64be7","Type":"ContainerDied","Data":"3e79533a6262db056d428579747c46fa8a290fb7a54a522cb94819fb2d033cf3"} Oct 30 21:36:07.494732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.494590 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 21:36:07.519146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.519108 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.533652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.533622 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 21:36:07.537936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.537908 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.538045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.537945 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.538115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.538085 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.538217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.538196 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.592436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.592400 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-991335-serving-cert\"" Oct 30 21:36:07.603720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.601860 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:36:07.604012 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.603985 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:36:07.639163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.638781 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.639163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.638859 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.639163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.638909 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.639163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.638929 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.639427 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:07.639294 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-serving-cert: secret "maas-api-e2e-worker-w5-991335-serving-cert" not found Oct 30 21:36:07.639427 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:07.639342 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls podName:94a2d290-6451-4544-a1e0-be827433e06f nodeName:}" failed. No retries permitted until 2026-10-30 21:36:08.139327258 +0000 UTC m=+1348.832809472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls") pod "maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" (UID: "94a2d290-6451-4544-a1e0-be827433e06f") : secret "maas-api-e2e-worker-w5-991335-serving-cert" not found Oct 30 21:36:07.639970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.639948 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.642289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.642270 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.728412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.728370 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:07.730653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.730626 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:07.939763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:07.932201 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:08.143741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.141409 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:08.146451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.146420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:08.148917 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.148896 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:08.230519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.230140 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:36:08.247528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.246952 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\" (UniqueName: \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\") pod \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " Oct 30 21:36:08.247528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.247049 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\") pod \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " Oct 30 21:36:08.247528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.247072 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\") pod \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " Oct 30 21:36:08.247528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.247125 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\") pod \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\" (UID: \"c20f97ea-11d1-439a-b9ad-4a2218f64be7\") " Oct 30 21:36:08.248067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.248039 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle" pod "c20f97ea-11d1-439a-b9ad-4a2218f64be7" (UID: "c20f97ea-11d1-439a-b9ad-4a2218f64be7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:08.248259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.248232 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle" pod "c20f97ea-11d1-439a-b9ad-4a2218f64be7" (UID: "c20f97ea-11d1-439a-b9ad-4a2218f64be7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:08.248482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.248460 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path" pod "c20f97ea-11d1-439a-b9ad-4a2218f64be7" (UID: "c20f97ea-11d1-439a-b9ad-4a2218f64be7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:08.249777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.249756 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s" pod "c20f97ea-11d1-439a-b9ad-4a2218f64be7" (UID: "c20f97ea-11d1-439a-b9ad-4a2218f64be7"). InnerVolumeSpecName "kube-api-access-k2t5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:08.348414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.348379 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2t5s\" (UniqueName: \"kubernetes.io/projected/c20f97ea-11d1-439a-b9ad-4a2218f64be7-kube-api-access-k2t5s\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:08.348414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.348410 4918 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-service-ca-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:08.348414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.348422 4918 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c20f97ea-11d1-439a-b9ad-4a2218f64be7-archives-path\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:08.348414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.348431 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20f97ea-11d1-439a-b9ad-4a2218f64be7-trusted-ca-bundle\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:08.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.437107 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:08.581596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.581125 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:36:08.581596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.581192 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:36:08.838885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.838797 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" Oct 30 21:36:08.838885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.838815 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rc98d-vb8mz" event={"ID":"c20f97ea-11d1-439a-b9ad-4a2218f64be7","Type":"ContainerDied","Data":"0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab"} Oct 30 21:36:08.838885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.838858 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1da88c0e6543f57ad75a799a7ad2b8ee710b270892339aa3cbf49149335eab" Oct 30 21:36:08.901419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.901368 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 21:36:08.921597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:08.919106 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:36:09.584731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:09.579187 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:36:09.584731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:09.579258 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:36:09.849845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:09.849737 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" event={"ID":"94a2d290-6451-4544-a1e0-be827433e06f","Type":"ContainerStarted","Data":"f0e35f5b27172cf078d734b103395bf4abb136d1cdf49f3b43a7357ba2ac737c"} Oct 30 21:36:09.924413 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:09.924346 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:10.120734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:10.120491 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:10.579112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:10.579022 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:36:10.579385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:10.579358 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:36:10.858035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:10.857947 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" event={"ID":"94a2d290-6451-4544-a1e0-be827433e06f","Type":"ContainerStarted","Data":"46febd0b44c6344fb00cdcad360fc8ae3696bf1da65067edb6f917b1dfc4513c"} Oct 30 21:36:10.858035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:10.858011 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:11.598419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:11.598392 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:36:11.598808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:11.598785 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:36:11.599553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:11.599536 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 21:36:11.830202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:11.830156 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" podStartSLOduration=3.8117471160000003 podStartE2EDuration="4.830143401s" totalImagesPullingTime="1.018396285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:10.994975058 +0000 UTC m=+1351.688457297" watchObservedRunningTime="2026-10-30 21:36:11.830143401 +0000 UTC m=+1352.523625661" Oct 30 21:36:11.953677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:11.953626 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:12.642555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.642521 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:12.867884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.867851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:12.942694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.942663 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 21:36:12.943308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.943293 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c20f97ea-11d1-439a-b9ad-4a2218f64be7" containerName="insights-gathering" Oct 30 21:36:12.943308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.943309 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20f97ea-11d1-439a-b9ad-4a2218f64be7" containerName="insights-gathering" Oct 30 21:36:12.943419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.943399 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20f97ea-11d1-439a-b9ad-4a2218f64be7" containerName="insights-gathering" Oct 30 21:36:12.946867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:12.946846 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.026318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.026285 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.026694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.026334 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.026694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.026576 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2gv\" (UniqueName: \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.026694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.026637 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.058851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.058825 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 21:36:13.092667 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.092634 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-afbe5b-serving-cert\"" Oct 30 21:36:13.128264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.128237 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2gv\" (UniqueName: \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.128406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.128293 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.128406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.128373 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.128519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.128407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.128563 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:13.128410 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-serving-cert: secret "maas-api-e2e-worker-w0-afbe5b-serving-cert" not found Oct 30 21:36:13.128612 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:13.128595 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls podName:cbad3c9f-1581-454f-9b7e-d3c6c7580300 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:13.628571065 +0000 UTC m=+1354.322053286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls") pod "maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" (UID: "cbad3c9f-1581-454f-9b7e-d3c6c7580300") : secret "maas-api-e2e-worker-w0-afbe5b-serving-cert" not found Oct 30 21:36:13.129236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.129209 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.130899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.130876 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.193367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.193286 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2gv\" (UniqueName: \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.201662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.201630 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:13.397751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.397663 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:13.631655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.631606 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:13.634797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.634767 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.637336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.637307 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") pod \"maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:13.860815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:13.860775 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:14.244450 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:36:14.244414 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbad3c9f_1581_454f_9b7e_d3c6c7580300.slice/crio-370a0e80650dddd69ed02136f8ace7a2201e69fa16cb6441f9357a07a83c2f18 WatchSource:0}: Error finding container 370a0e80650dddd69ed02136f8ace7a2201e69fa16cb6441f9357a07a83c2f18: Status 404 returned error can't find the container with id 370a0e80650dddd69ed02136f8ace7a2201e69fa16cb6441f9357a07a83c2f18 Oct 30 21:36:14.248343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:14.248315 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 21:36:14.886622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:14.886591 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" event={"ID":"cbad3c9f-1581-454f-9b7e-d3c6c7580300","Type":"ContainerStarted","Data":"370a0e80650dddd69ed02136f8ace7a2201e69fa16cb6441f9357a07a83c2f18"} Oct 30 21:36:15.802777 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:15.802742 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:36:15.895623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:15.895587 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" event={"ID":"cbad3c9f-1581-454f-9b7e-d3c6c7580300","Type":"ContainerStarted","Data":"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389"} Oct 30 21:36:15.895823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:15.895655 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:16.019422 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.019360 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" podStartSLOduration=3.049316831 podStartE2EDuration="4.01934644s" totalImagesPullingTime="970.029609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:16.016235758 +0000 UTC m=+1356.709717990" watchObservedRunningTime="2026-10-30 21:36:16.01934644 +0000 UTC m=+1356.712828678" Oct 30 21:36:16.193350 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.193312 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:16.253142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.253107 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:35326->10.132.0.64:15021: read: connection reset by peer" start-of-body= Oct 30 21:36:16.253262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.253175 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:35326->10.132.0.64:15021: read: connection reset by peer" Oct 30 21:36:16.871831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.871798 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 21:36:16.904217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.904190 4918 generic.go:356] "Generic (PLEG): container finished" podID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerID="2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b" exitCode=0 Oct 30 21:36:16.904381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.904274 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerDied","Data":"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b"} Oct 30 21:36:16.904795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.904778 4918 scope.go:122] "RemoveContainer" containerID="2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b" Oct 30 21:36:16.906107 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.906082 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:16.906187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.906148 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:16.906187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:16.906180 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:17.913985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:17.913943 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerStarted","Data":"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5"} Oct 30 21:36:17.932803 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:17.932700 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:36:17.932950 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:17.932881 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:17.932950 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:17.932932 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:18.132734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:18.132677 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:18.932411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:18.932378 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:18.932939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:18.932438 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:19.932015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:19.931975 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:19.932196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:19.932029 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:20.932214 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:20.932185 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:20.932680 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:20.932239 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:21.911211 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:21.911185 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 21:36:21.932055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:21.932029 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:21.932202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:21.932074 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:22.561855 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.561826 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 21:36:22.565954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.565935 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.619398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.619372 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-71aab5-serving-cert\"" Oct 30 21:36:22.624014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.623969 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 21:36:22.646172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.646142 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpwg\" (UniqueName: \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.646313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.646192 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.646313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.646211 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.646427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.646337 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.747105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.747073 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpwg\" (UniqueName: \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.747284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.747124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.747284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.747144 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.747284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.747170 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.747460 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:22.747316 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-serving-cert: secret "maas-api-e2e-worker-w3-71aab5-serving-cert" not found Oct 30 21:36:22.747460 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:22.747379 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls podName:f21fca56-4a41-4e0a-880e-7380c80c2219 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:23.247356848 +0000 UTC m=+1363.940839082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls") pod "maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" (UID: "f21fca56-4a41-4e0a-880e-7380c80c2219") : secret "maas-api-e2e-worker-w3-71aab5-serving-cert" not found Oct 30 21:36:22.748019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.747992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.749650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.749628 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.801379 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.801217 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:22.820492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.820408 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpwg\" (UniqueName: \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:22.932450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.932418 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:36:22.932627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:22.932481 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:36:23.003697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.003609 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:23.224980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.224930 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:23.253301 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.253271 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:23.255645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.255626 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:23.502738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.502637 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:23.880470 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:36:23.880436 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21fca56_4a41_4e0a_880e_7380c80c2219.slice/crio-292ddd0a98d6dc6df2f7d4ff0fc5e8daa559c89982e289d7da22f1b1a24fde59 WatchSource:0}: Error finding container 292ddd0a98d6dc6df2f7d4ff0fc5e8daa559c89982e289d7da22f1b1a24fde59: Status 404 returned error can't find the container with id 292ddd0a98d6dc6df2f7d4ff0fc5e8daa559c89982e289d7da22f1b1a24fde59 Oct 30 21:36:23.882887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.882853 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 21:36:23.935923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.935894 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:36:23.936182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.936162 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:36:23.936877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.936858 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:36:23.953524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:23.953497 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" event={"ID":"f21fca56-4a41-4e0a-880e-7380c80c2219","Type":"ContainerStarted","Data":"292ddd0a98d6dc6df2f7d4ff0fc5e8daa559c89982e289d7da22f1b1a24fde59"} Oct 30 21:36:24.961310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:24.961280 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" event={"ID":"f21fca56-4a41-4e0a-880e-7380c80c2219","Type":"ContainerStarted","Data":"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499"} Oct 30 21:36:25.083141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:25.083054 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" podStartSLOduration=2.232117753 podStartE2EDuration="3.083039812s" totalImagesPullingTime="850.922059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:25.081858189 +0000 UTC m=+1365.775340424" watchObservedRunningTime="2026-10-30 21:36:25.083039812 +0000 UTC m=+1365.776522047" Oct 30 21:36:25.211888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:25.211851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:25.413441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:25.413410 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:25.967364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:25.967331 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:28.124663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:28.124624 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:28.341470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:28.341439 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:28.814918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:28.814886 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:29.021544 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:29.021505 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:31.999071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:31.999039 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 21:36:33.187736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:33.178272 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:37.021764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:37.021683 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:38.250913 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:38.250874 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:38.709754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:38.709666 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:39.062924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.062896 4918 generic.go:356] "Generic (PLEG): container finished" podID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerID="03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75" exitCode=0 Oct 30 21:36:39.063096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.062975 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerDied","Data":"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75"} Oct 30 21:36:39.063340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.063325 4918 scope.go:122] "RemoveContainer" containerID="03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75" Oct 30 21:36:39.064641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.064617 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:39.064699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.064682 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:39.064758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.064730 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:39.333754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.333658 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:39.537701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:39.537657 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:40.073730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:40.072195 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerStarted","Data":"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa"} Oct 30 21:36:40.194735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:40.194460 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:40.223105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:40.220983 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:36:40.223105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:40.221296 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:36:40.223105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:40.221347 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:36:41.222008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:41.221976 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:36:41.222406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:41.222027 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:36:41.711730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:41.695072 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:42.221786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:42.221751 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:36:42.221980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:42.221812 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:36:43.221775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:43.221735 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:36:43.222168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:43.221785 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:36:43.366063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:43.365967 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:44.221900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:44.221865 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:36:44.222288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:44.221917 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 21:36:45.225361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:45.225331 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:36:45.225796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:45.225611 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:36:45.226415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:45.226396 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:36:45.803653 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:36:45.803621 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:36:48.040550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:48.040510 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.247922 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:48.247890 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.704743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:48.704687 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.889715 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:48.889664 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:52.917621 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:52.917561 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:55.353931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:55.353871 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.054721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:56.054682 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.256478 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:56.256441 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.362197 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:57.362160 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.832216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:36:57.832124 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.223929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:00.223881 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.852066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:00.852033 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.058940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:01.058899 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:02.631000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:02.630960 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.253400 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:07.253362 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:09.534185 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:09.534150 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:10.773666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:10.773620 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:11.430193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:11.430156 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:11.646175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:11.646142 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:11.843741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:11.843650 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.033907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:12.033872 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.761677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:12.761622 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.801338 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:37:15.801303 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:37:16.714090 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:16.714055 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.309348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:21.309308 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:25.902041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:25.902002 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:29.566195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:29.566158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:30.523420 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:30.523385 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:36.398290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:36.398255 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:37.372034 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:37.371997 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:39.018906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:39.018528 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:39.691787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:39.691750 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.302062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:44.302024 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.708594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:44.708561 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:45.805370 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:37:45.805332 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:37:48.827258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:48.827215 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:53.410966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:53.410921 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.916489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:57.916453 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:58.926440 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:37:58.926343 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:02.541024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:02.540995 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:07.146596 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:07.146558 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:09.835554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:09.835516 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:11.632734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:11.632686 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:15.799997 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:38:15.799951 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:38:16.219105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:16.219056 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:20.827413 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:20.827328 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:25.392622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:25.392574 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.072654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:26.072613 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.258938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:26.258901 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:30.000477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:30.000422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.509610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:34.509569 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.725197 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:37.725161 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:39.096098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.096063 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:39.457882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.457839 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:39.596623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.596593 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:38:39.600179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.600154 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:38:39.612095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.612066 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:38:39.615774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.615753 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:38:39.618113 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.618096 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:38:39.621433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:39.621414 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:38:41.798069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:41.798020 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:43.701585 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:43.701544 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:45.802864 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:38:45.802830 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:38:48.220449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:48.220398 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.723718 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:52.723665 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.306042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:57.306004 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.822879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.822845 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:38:58.826898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.826878 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.880608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.880583 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-554f46-openshift-default-dockercfg-lpld4\"" Oct 30 21:38:58.936499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.936468 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:38:58.958030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.957998 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958119 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958154 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958277 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958233 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958315 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958278 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chfq\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958347 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958410 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958511 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:58.958669 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:58.958570 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060018 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060082 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060175 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060235 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060262 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060378 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060301 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060378 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060341 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chfq\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060422 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060456 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060458 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060495 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060687 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060657 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.060893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060872 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.061000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.060974 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.062609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.062585 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.062762 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.062743 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.122345 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.122316 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.123425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.123398 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chfq\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.140202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.140182 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:38:59.566048 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:38:59.566016 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403df095_8f5d_4087_a442_d6ea292c4b3e.slice/crio-b49703c7868e7cb3b759feac9120501ab3c0d6232577e710c8bbf86e72b14437 WatchSource:0}: Error finding container b49703c7868e7cb3b759feac9120501ab3c0d6232577e710c8bbf86e72b14437: Status 404 returned error can't find the container with id b49703c7868e7cb3b759feac9120501ab3c0d6232577e710c8bbf86e72b14437 Oct 30 21:38:59.568276 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.568243 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:59.568385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.568310 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:59.568385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.568337 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:59.569170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:38:59.569150 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:39:00.045258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.045214 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" event={"ID":"403df095-8f5d-4087-a442-d6ea292c4b3e","Type":"ContainerStarted","Data":"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614"} Oct 30 21:39:00.045770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.045280 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" event={"ID":"403df095-8f5d-4087-a442-d6ea292c4b3e","Type":"ContainerStarted","Data":"b49703c7868e7cb3b759feac9120501ab3c0d6232577e710c8bbf86e72b14437"} Oct 30 21:39:00.140917 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.140888 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:39:00.142663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.142635 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:00.142795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.142684 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:00.186729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:00.179779 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podStartSLOduration=2.179760693 podStartE2EDuration="2.179760693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:00.16719435 +0000 UTC m=+1520.860676598" watchObservedRunningTime="2026-10-30 21:39:00.179760693 +0000 UTC m=+1520.873242930" Oct 30 21:39:01.141000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:01.140968 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:01.141459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:01.141024 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:01.709522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:01.709485 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:01.932553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:01.932502 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:02.141439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:02.141403 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:02.141855 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:02.141454 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:03.140683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:03.140650 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:03.140871 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:03.140731 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:04.140685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:04.140644 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:04.141124 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:04.140737 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:05.140757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:05.140727 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:39:05.141164 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:05.140776 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:39:06.160329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:06.160291 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:39:06.161303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:06.161279 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:39:06.162949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:06.162928 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:39:06.628576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:06.628540 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:09.107522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:09.107486 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:11.325454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:11.325416 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:15.801339 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:39:15.801298 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:39:15.922003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:15.921966 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:20.498973 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:20.498933 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:25.093901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:25.093850 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:27.341098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:27.341063 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:29.642644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:29.642598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.212512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:34.212479 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.165408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:38.165374 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.365389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:38.365356 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.840894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:38.840858 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:43.435379 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:43.435342 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.800614 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:39:45.800577 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:39:47.965595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:47.965563 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:52.630735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:52.626352 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:57.231819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:57.231787 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:59.583829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:59.583797 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:39:59.584289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:39:59.584042 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" containerID="cri-o://25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9" gracePeriod=30 Oct 30 21:40:00.070528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.070487 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:40:00.074666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.074642 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.131052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.131022 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a2dced-openshift-default-dockercfg-ll74c\"" Oct 30 21:40:00.136152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.136124 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:40:00.173575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173552 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173585 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173749 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173694 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173789 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173773 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173886 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173929 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173947 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.173986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.173966 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9rk\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274664 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274633 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274671 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274807 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274846 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274879 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9rk\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.274965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274941 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274979 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.274991 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275057 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275110 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275151 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275181 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275225 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.275720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.275688 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.277128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.277105 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.277355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.277328 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.336887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.336836 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.336998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.336978 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9rk\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.396127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.396105 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:00.824464 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:40:00.824433 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae45481e_43e1_4ba5_8ac6_5bfade5ae1b9.slice/crio-829d6933bd57ab2fc8c623f70b81e7f3638840e39e46bf742b40f1c29b0b6c78 WatchSource:0}: Error finding container 829d6933bd57ab2fc8c623f70b81e7f3638840e39e46bf742b40f1c29b0b6c78: Status 404 returned error can't find the container with id 829d6933bd57ab2fc8c623f70b81e7f3638840e39e46bf742b40f1c29b0b6c78 Oct 30 21:40:00.827220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.827186 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:00.827344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.827264 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:00.827344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.827312 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:00.827757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:00.827729 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:40:01.511027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:01.510982 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" event={"ID":"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9","Type":"ContainerStarted","Data":"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14"} Oct 30 21:40:01.511237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:01.511042 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" event={"ID":"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9","Type":"ContainerStarted","Data":"829d6933bd57ab2fc8c623f70b81e7f3638840e39e46bf742b40f1c29b0b6c78"} Oct 30 21:40:01.699430 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:01.699375 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podStartSLOduration=1.6993593310000001 podStartE2EDuration="1.699359331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:01.695628337 +0000 UTC m=+1582.389110573" watchObservedRunningTime="2026-10-30 21:40:01.699359331 +0000 UTC m=+1582.392841567" Oct 30 21:40:01.892513 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:01.892476 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:02.398317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:02.398284 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:02.400042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:02.400016 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.400167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:02.400062 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:40:03.397269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:03.397231 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.397816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:03.397779 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:40:04.397183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.397141 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:40:04.397376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.397197 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:40:04.868523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.868499 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:40:04.928616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928586 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928639 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928692 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928753 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928813 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928867 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.928931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928892 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:04.929087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928965 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.929087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.928988 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:04.929087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929036 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:04.929240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929143 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:04.929559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929489 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:04.929559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929516 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:04.929559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929534 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:04.929884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.929771 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:04.931502 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.931477 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:04.931736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.931697 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:40:04.931806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.931728 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "kube-api-access-hdmsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:04.931806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:04.931787 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:05.030110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030041 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\") pod \"03581726-86b6-4ad3-b334-8161899bf209\" (UID: \"03581726-86b6-4ad3-b334-8161899bf209\") " Oct 30 21:40:05.030252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030239 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmsv\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-kube-api-access-hdmsv\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.030297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030255 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03581726-86b6-4ad3-b334-8161899bf209-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.030297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030265 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03581726-86b6-4ad3-b334-8161899bf209-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.030297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030256 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket" pod "03581726-86b6-4ad3-b334-8161899bf209" (UID: "03581726-86b6-4ad3-b334-8161899bf209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:05.030297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030274 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03581726-86b6-4ad3-b334-8161899bf209-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.030448 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.030316 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.131688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.131660 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03581726-86b6-4ad3-b334-8161899bf209-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.397190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.397156 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:40:05.397453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.397427 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:40:05.540885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.540848 4918 generic.go:356] "Generic (PLEG): container finished" podID="03581726-86b6-4ad3-b334-8161899bf209" containerID="25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9" exitCode=0 Oct 30 21:40:05.541083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.540894 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" Oct 30 21:40:05.541083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.540923 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" event={"ID":"03581726-86b6-4ad3-b334-8161899bf209","Type":"ContainerDied","Data":"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9"} Oct 30 21:40:05.541083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.540976 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s" event={"ID":"03581726-86b6-4ad3-b334-8161899bf209","Type":"ContainerDied","Data":"6af5055c800a18a4a4490e4dec32a2de7523952a10f277d87bc58169a2af9b3a"} Oct 30 21:40:05.541083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.541001 4918 scope.go:122] "RemoveContainer" containerID="25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9" Oct 30 21:40:05.551234 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.551225 4918 scope.go:122] "RemoveContainer" containerID="25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9" Oct 30 21:40:05.551491 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:40:05.551470 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9\": container with ID starting with 25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9 not found: ID does not exist" containerID="25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9" Oct 30 21:40:05.551556 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.551497 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9"} err="failed to get container status \"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9\": rpc error: code = NotFound desc = could not find container \"25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9\": container with ID starting with 25f9987cbe48127066d84cd9cbff26de9273cbe6875ef3263d7141aae446b1c9 not found: ID does not exist" Oct 30 21:40:05.714086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.713990 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:40:05.718822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:05.718801 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kd64s"] Oct 30 21:40:06.397196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:06.397158 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:40:06.397366 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:06.397210 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:40:06.595105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:06.595067 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.429688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:07.429647 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03581726-86b6-4ad3-b334-8161899bf209" path="/var/lib/kubelet/pods/03581726-86b6-4ad3-b334-8161899bf209/volumes" Oct 30 21:40:07.512844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:07.512801 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:07.513430 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:07.513401 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:07.515457 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:07.515433 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:40:11.446083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:11.446038 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.810216 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:40:15.810174 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:40:16.236058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.236026 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:16.496402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.496318 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:40:16.497162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.497140 4918 cpu_manager.go:414] "containerMap: removing container" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" Oct 30 21:40:16.497216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.497166 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" Oct 30 21:40:16.497375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.497362 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="03581726-86b6-4ad3-b334-8161899bf209" containerName="istio-proxy" Oct 30 21:40:16.502894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.502871 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.566439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.566415 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-991335-openshift-default-dockercfg-xsdjt\"" Oct 30 21:40:16.581130 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581106 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcxn\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581220 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581256 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581278 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581376 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581436 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581533 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.581611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.581558 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.621078 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.621030 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:40:16.682771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.682700 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.682901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.682825 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.682988 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.682924 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.682988 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.682956 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.682987 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcxn\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683001 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683100 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683147 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683179 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683208 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683231 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683403 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683526 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.683858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.683840 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.685420 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.685393 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.685554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.685533 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.745732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.745675 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.746102 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.746081 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcxn\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:16.820383 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:16.820328 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:17.251367 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:40:17.251332 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ab9c7b_871b_46a9_be09_022d3fe9db5f.slice/crio-a4d3ce2f1a4c89ef5b1635007defa981567ca7806d52e4931d93da8202552213 WatchSource:0}: Error finding container a4d3ce2f1a4c89ef5b1635007defa981567ca7806d52e4931d93da8202552213: Status 404 returned error can't find the container with id a4d3ce2f1a4c89ef5b1635007defa981567ca7806d52e4931d93da8202552213 Oct 30 21:40:17.254552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.254514 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:17.254650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.254606 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:17.254650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.254642 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:17.255604 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.255581 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:40:17.653764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.653721 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" event={"ID":"04ab9c7b-871b-46a9-be09-022d3fe9db5f","Type":"ContainerStarted","Data":"c6ebaff5ee6c22f54e6af10be112f7e9faea87f69d6eff913817e80d34f69dd4"} Oct 30 21:40:17.653977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.653776 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" event={"ID":"04ab9c7b-871b-46a9-be09-022d3fe9db5f","Type":"ContainerStarted","Data":"a4d3ce2f1a4c89ef5b1635007defa981567ca7806d52e4931d93da8202552213"} Oct 30 21:40:17.776970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.776910 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podStartSLOduration=1.77689284 podStartE2EDuration="1.77689284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:17.776088439 +0000 UTC m=+1598.469570676" watchObservedRunningTime="2026-10-30 21:40:17.77689284 +0000 UTC m=+1598.470375078" Oct 30 21:40:17.820537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:17.820508 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:18.822719 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:18.822684 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:18.823175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:18.822758 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:19.820832 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:19.820795 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:19.821094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:19.820859 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:20.821201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:20.821168 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:20.821593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:20.821221 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:21.133500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:21.133458 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:21.820943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:21.820910 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:21.821129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:21.820998 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:22.820793 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:22.820758 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:22.821254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:22.820828 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:23.821221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:23.821191 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:23.821617 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:23.821242 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:40:24.824755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:24.824719 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:24.825238 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:24.825129 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:24.825892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:24.825870 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:40:25.722978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:25.722943 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:29.597846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.597799 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:29.820887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.820855 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:40:29.824856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.824836 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.874772 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.874723 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:40:29.885227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.883900 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:29.894040 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.894013 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:40:29.936469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.936433 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.936787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.936756 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.936952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.936904 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.936968 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937085 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937052 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmv2\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937139 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937125 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937174 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937223 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937301 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937260 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:29.937376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:29.937352 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:40:30.037947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.037914 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.037962 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038001 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038042 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038147 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038200 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038235 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038267 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038316 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzb5r\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038364 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038376 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038421 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038475 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038515 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038519 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038545 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038604 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038626 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038662 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038744 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038814 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmv2\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.038966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.038873 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.039643 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.039618 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.041294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.041271 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.041576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.041554 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.102571 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.102544 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.102664 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.102606 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmv2\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\") pod \"maas-default-gateway-openshift-default-67bf986f99-lq6p6\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.139470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139413 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:30.139763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139740 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139795 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139821 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139851 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzb5r\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139876 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139911 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.139952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139944 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.139978 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140007 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140302 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140399 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140489 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140562 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140501 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.140811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.140789 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.142227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.142199 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.142331 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.142200 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.209333 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.209301 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.213022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.212962 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzb5r\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\") pod \"maas-default-gateway-openshift-default-67bf986f99-4fgdq\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.330998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.326692 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:30.500377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.500285 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:30.571303 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:40:30.571196 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70df8927_e3f6_4081_9fa4_8170096e7664.slice/crio-200faac263d13f02dc0ab9aacfe1fdf4317aecc396350804a9fcc3058b7ae087 WatchSource:0}: Error finding container 200faac263d13f02dc0ab9aacfe1fdf4317aecc396350804a9fcc3058b7ae087: Status 404 returned error can't find the container with id 200faac263d13f02dc0ab9aacfe1fdf4317aecc396350804a9fcc3058b7ae087 Oct 30 21:40:30.573932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.573889 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.574026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.573975 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.574026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.574016 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.574228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.574207 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:40:30.762739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.762625 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" event={"ID":"70df8927-e3f6-4081-9fa4-8170096e7664","Type":"ContainerStarted","Data":"3202651ea4e6f2c0647c9e18fb7209852b7bff5e7a7fb6ab267b39954b073b6c"} Oct 30 21:40:30.762739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.762685 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" event={"ID":"70df8927-e3f6-4081-9fa4-8170096e7664","Type":"ContainerStarted","Data":"200faac263d13f02dc0ab9aacfe1fdf4317aecc396350804a9fcc3058b7ae087"} Oct 30 21:40:30.874996 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:40:30.874967 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e7593_1fde_41c9_aa03_8a7827b29571.slice/crio-1bd6382a08b52eb10c996f87de395931b26cd33a475cadeaf6360054c0f30847 WatchSource:0}: Error finding container 1bd6382a08b52eb10c996f87de395931b26cd33a475cadeaf6360054c0f30847: Status 404 returned error can't find the container with id 1bd6382a08b52eb10c996f87de395931b26cd33a475cadeaf6360054c0f30847 Oct 30 21:40:30.877659 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.877624 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.877806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.877721 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.877806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.877771 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:30.880187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.880165 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:40:30.887942 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:30.887891 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podStartSLOduration=1.887868369 podStartE2EDuration="1.887868369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:30.885423531 +0000 UTC m=+1611.578905780" watchObservedRunningTime="2026-10-30 21:40:30.887868369 +0000 UTC m=+1611.581350607" Oct 30 21:40:31.041377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.041289 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:40:31.046003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.045977 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.100878 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.100842 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-afbe5b-openshift-default-dockercfg-jbdjs\"" Oct 30 21:40:31.123121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.123061 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:40:31.140026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.139998 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:31.141126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.141100 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:31.141233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.141155 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:31.152777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.152750 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.152890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.152793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.152992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.152962 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153027 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153106 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153153 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153143 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153176 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqfh\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153256 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.153323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.153308 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.255676 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.255761 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.255837 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.255886 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.255955 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.256003 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.256043 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqfh\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.256089 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.256816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.256137 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.257419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.257278 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.257419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.257319 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.257586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.257550 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.257881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.257859 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.258386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.258360 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.260203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.260181 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.260753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.260734 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.318479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.318398 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.318479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.318415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqfh\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.376739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.375924 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:31.803731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.803665 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" event={"ID":"608e7593-1fde-41c9-aa03-8a7827b29571","Type":"ContainerStarted","Data":"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f"} Oct 30 21:40:31.804216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.803759 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" event={"ID":"608e7593-1fde-41c9-aa03-8a7827b29571","Type":"ContainerStarted","Data":"1bd6382a08b52eb10c996f87de395931b26cd33a475cadeaf6360054c0f30847"} Oct 30 21:40:31.835565 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:40:31.835536 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdb39ff_d4e2_4023_a65e_c15faa8d54a2.slice/crio-54487c22633f40d4a7380f3df39f78b9af8c651d2a1260041f431b588373997e WatchSource:0}: Error finding container 54487c22633f40d4a7380f3df39f78b9af8c651d2a1260041f431b588373997e: Status 404 returned error can't find the container with id 54487c22633f40d4a7380f3df39f78b9af8c651d2a1260041f431b588373997e Oct 30 21:40:31.836777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.836753 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:40:31.839069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.839038 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:31.839145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.839102 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:31.839145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.839128 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:31.932731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:31.930018 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podStartSLOduration=2.929999334 podStartE2EDuration="2.929999334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:31.929896799 +0000 UTC m=+1612.623379019" watchObservedRunningTime="2026-10-30 21:40:31.929999334 +0000 UTC m=+1612.623481571" Oct 30 21:40:32.143622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.143586 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:32.143892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.143864 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:32.514788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.513813 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:32.514788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.513944 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:32.514788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.513985 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:32.864741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.862057 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" event={"ID":"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2","Type":"ContainerStarted","Data":"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702"} Oct 30 21:40:32.864741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:32.862135 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" event={"ID":"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2","Type":"ContainerStarted","Data":"54487c22633f40d4a7380f3df39f78b9af8c651d2a1260041f431b588373997e"} Oct 30 21:40:33.021740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.018899 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podStartSLOduration=3.018878917 podStartE2EDuration="3.018878917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:33.016929904 +0000 UTC m=+1613.710412141" watchObservedRunningTime="2026-10-30 21:40:33.018878917 +0000 UTC m=+1613.712361153" Oct 30 21:40:33.145783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.145159 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:33.145783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.145222 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:33.390738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.388864 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:33.394729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.394529 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:33.394729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.394586 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:33.504788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.504129 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:33.504788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:33.504196 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:34.146508 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.146469 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:34.147498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.147087 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:34.397845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.389667 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:34.397845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.390269 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:34.397845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.390318 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:34.507735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.502298 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:34.508034 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.507997 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:34.600731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:34.600393 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:35.141183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.141144 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:35.141477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.141447 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:35.399786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.391950 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:35.399786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.392013 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:35.502731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.502048 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:35.502731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.502107 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:35.657568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:35.657206 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:36.146729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.145876 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:36.146729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.145938 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:36.387126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.387088 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:36.387411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.387379 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:36.509782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.501420 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:36.509782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:36.501495 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:37.140801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.140766 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:37.140989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.140823 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:37.389101 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.389062 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:37.390116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.390084 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:37.503787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.503361 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:37.503787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:37.503422 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:38.140623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.140593 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:38.141132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.141098 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:40:38.386130 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.386098 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:38.386307 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.386153 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:40:38.403683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.403607 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:38.500775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.500742 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:38.500955 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:38.500797 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:40:40.081731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.062125 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:40.095735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.090112 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:40.158736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.158077 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:40:40.226730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.212796 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:40.226730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.212871 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:40.555369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.555327 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4fgdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:40.772731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:40.764934 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:41.182510 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.182087 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:41.183381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.183351 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:41.184926 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.184898 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:40:41.214763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.214249 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lq6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:41.214763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.214307 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:41.273904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.273872 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:41.275239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.275206 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:41.276007 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.275960 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:40:41.477295 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:41.477210 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:45.804676 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:40:45.804638 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:40:46.103983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:46.103851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:48.606882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:48.606852 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:48.802166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:48.802130 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:50.655744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:50.655662 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:55.236550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:55.236512 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:59.776968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.776935 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:40:59.781371 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.781337 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.798127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.798069 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:59.840624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.840592 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:40:59.924595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924559 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.924595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924599 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.924821 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924804 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.924859 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924830 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.924859 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924851 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.924997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.924982 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.925035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.925004 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d557x\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.925068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.925042 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:40:59.925101 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:40:59.925079 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026176 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026207 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026227 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026287 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026303 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d557x\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026352 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026380 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026422 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026455 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026634 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026694 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026802 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.026898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.026863 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.027159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.027098 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.028661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.028638 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.028928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.028911 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.088369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.088348 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.088484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.088444 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d557x\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.115371 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.115343 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:00.547479 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:41:00.547447 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0786293a_a84a_4060_a771_2adbf773da4e.slice/crio-acb6f9cde6e5409e655d5cbadfa6c9a6e5aeab4ab623ab51d0092f57890c7139 WatchSource:0}: Error finding container acb6f9cde6e5409e655d5cbadfa6c9a6e5aeab4ab623ab51d0092f57890c7139: Status 404 returned error can't find the container with id acb6f9cde6e5409e655d5cbadfa6c9a6e5aeab4ab623ab51d0092f57890c7139 Oct 30 21:41:00.549887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.549851 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:00.549992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.549932 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:00.549992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.549975 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:00.551598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.551575 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:41:00.565110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.565092 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:41:00.570738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.570717 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.654919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.654893 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:41:00.734326 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734290 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734387 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734461 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734502 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734557 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734617 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7x8\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734676 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734700 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.734830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.734755 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836261 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7x8\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836313 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836346 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836519 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836683 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836745 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836753 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.836840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836800 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836851 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836918 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836922 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.836951 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.837293 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.837615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.837594 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.838999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.838975 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.839233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.839216 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.899057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.899028 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7x8\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.899189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.899071 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:00.922432 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:00.922233 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:01.389609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.389572 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:41:01.403376 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:41:01.403321 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9cec11_4398_49ad_b891_54d5450f1624.slice/crio-be49689f9c19abf2d4489eea2253977577cbc505f61e0df58611013a5dc2f3e6 WatchSource:0}: Error finding container be49689f9c19abf2d4489eea2253977577cbc505f61e0df58611013a5dc2f3e6: Status 404 returned error can't find the container with id be49689f9c19abf2d4489eea2253977577cbc505f61e0df58611013a5dc2f3e6 Oct 30 21:41:01.406467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.406430 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:01.406561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.406525 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:01.406601 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.406567 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:01.551676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.551633 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" event={"ID":"db9cec11-4398-49ad-b891-54d5450f1624","Type":"ContainerStarted","Data":"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5"} Oct 30 21:41:01.551857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.551717 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" event={"ID":"db9cec11-4398-49ad-b891-54d5450f1624","Type":"ContainerStarted","Data":"be49689f9c19abf2d4489eea2253977577cbc505f61e0df58611013a5dc2f3e6"} Oct 30 21:41:01.553863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.553829 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" event={"ID":"0786293a-a84a-4060-a771-2adbf773da4e","Type":"ContainerStarted","Data":"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923"} Oct 30 21:41:01.553981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.553887 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" event={"ID":"0786293a-a84a-4060-a771-2adbf773da4e","Type":"ContainerStarted","Data":"acb6f9cde6e5409e655d5cbadfa6c9a6e5aeab4ab623ab51d0092f57890c7139"} Oct 30 21:41:01.702976 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.702851 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podStartSLOduration=1.702829116 podStartE2EDuration="1.702829116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:01.692388067 +0000 UTC m=+1642.385870340" watchObservedRunningTime="2026-10-30 21:41:01.702829116 +0000 UTC m=+1642.396311354" Oct 30 21:41:01.934349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:01.934290 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:02.118167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:02.117040 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:02.135730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:02.133909 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:02.135730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:02.133974 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:02.931732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:02.929327 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:02.931732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:02.929392 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:03.120154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:03.120110 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:03.120779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:03.120701 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:03.931100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:03.931058 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:03.932131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:03.932099 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:04.117743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:04.117209 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:04.117743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:04.117286 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:04.928726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:04.928090 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:04.928726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:04.928188 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:05.129177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:05.129124 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.129384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:05.129190 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:05.317282 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:05.317118 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:05.929128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:05.929090 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.929760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:05.929681 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:06.120610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:06.120571 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.120931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:06.120895 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:06.934730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:06.934399 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.934730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:06.934460 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:07.116186 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:07.116146 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.116393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:07.116204 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:41:07.923028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:07.922992 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.923321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:07.923289 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:08.307114 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.307017 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:08.307971 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.307950 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:08.308796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.308776 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:41:08.491510 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.491450 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podStartSLOduration=9.491432659 podStartE2EDuration="9.491432659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:01.824152986 +0000 UTC m=+1642.517635223" watchObservedRunningTime="2026-10-30 21:41:08.491432659 +0000 UTC m=+1649.184914896" Oct 30 21:41:08.924206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.924167 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.924411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:08.924235 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 21:41:09.998825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:09.998792 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:10.611036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:10.611007 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:10.611503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:10.611474 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:10.612345 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:10.612324 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:41:14.615066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:14.615025 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:15.802846 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:41:15.802815 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:41:19.250273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:19.250236 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:22.253197 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:22.253121 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:23.911128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:23.911084 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:28.671239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:28.671200 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:30.035673 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.035634 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:30.496334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.496300 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:41:30.500499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.500480 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.590254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.590226 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:41:30.595500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595468 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595516 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595676 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595804 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595740 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595834 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clmz\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595876 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.595983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595936 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.596054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.595987 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.596054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.596046 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.696943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.696914 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.696952 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697007 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clmz\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697031 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697167 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697238 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697307 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697349 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697612 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697791 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697733 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697918 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.697985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.697962 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.698154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.698128 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.698355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.698333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.699811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.699791 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.700017 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.699999 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.758672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.758624 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clmz\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.762903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.762881 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:30.818698 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:30.818667 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:31.255224 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:41:31.251260 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4f1503_cd63_49b5_81f8_58c8290d79ed.slice/crio-ff8eeca9e3b4d9e4053f8ba116df82cac0db628cbcd9bab51c7331e29844823e WatchSource:0}: Error finding container ff8eeca9e3b4d9e4053f8ba116df82cac0db628cbcd9bab51c7331e29844823e: Status 404 returned error can't find the container with id ff8eeca9e3b4d9e4053f8ba116df82cac0db628cbcd9bab51c7331e29844823e Oct 30 21:41:31.257167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.257132 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.257280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.257208 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.257280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.257239 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.258169 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.258148 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:41:31.556054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.555972 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:41:31.561298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.561264 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.620335 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.620298 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:41:31.715733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.715686 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.715890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.715773 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.715890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.715861 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716052 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716197 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716176 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716228 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716350 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716334 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhzw\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716373 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.716439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.716399 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819026 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819092 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819163 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhzw\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819196 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819221 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819256 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819342 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.819778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819772 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.820326 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819824 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.820326 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.819953 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.820326 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.820303 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.820523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.820496 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.821058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.821030 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.821319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.821292 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.823203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.823176 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.823445 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.823427 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.882952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.882888 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhzw\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.885324 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.885275 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:31.916089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.915795 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" event={"ID":"3b4f1503-cd63-49b5-81f8-58c8290d79ed","Type":"ContainerStarted","Data":"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2"} Oct 30 21:41:31.916089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:31.915862 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" event={"ID":"3b4f1503-cd63-49b5-81f8-58c8290d79ed","Type":"ContainerStarted","Data":"ff8eeca9e3b4d9e4053f8ba116df82cac0db628cbcd9bab51c7331e29844823e"} Oct 30 21:41:32.041374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.041321 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podStartSLOduration=2.041307551 podStartE2EDuration="2.041307551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:32.038115364 +0000 UTC m=+1672.731597612" watchObservedRunningTime="2026-10-30 21:41:32.041307551 +0000 UTC m=+1672.734789852" Oct 30 21:41:32.184989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.184947 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:32.636025 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:41:32.635989 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee57187_4974_4a5d_b0f8_301a109e8f04.slice/crio-7e76d0d4140c4786269d34ff7fe258abd28cf5e3413a92f042c49ae6b11c3342 WatchSource:0}: Error finding container 7e76d0d4140c4786269d34ff7fe258abd28cf5e3413a92f042c49ae6b11c3342: Status 404 returned error can't find the container with id 7e76d0d4140c4786269d34ff7fe258abd28cf5e3413a92f042c49ae6b11c3342 Oct 30 21:41:32.639627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.639134 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.639627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.639227 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.639627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.639282 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.640118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.640085 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:41:32.819733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.819688 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:32.821010 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.820984 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:41:32.821081 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.821039 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:41:32.924012 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.923935 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" event={"ID":"cee57187-4974-4a5d-b0f8-301a109e8f04","Type":"ContainerStarted","Data":"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea"} Oct 30 21:41:32.924012 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:32.923999 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" event={"ID":"cee57187-4974-4a5d-b0f8-301a109e8f04","Type":"ContainerStarted","Data":"7e76d0d4140c4786269d34ff7fe258abd28cf5e3413a92f042c49ae6b11c3342"} Oct 30 21:41:33.057421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.057364 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podStartSLOduration=2.057350716 podStartE2EDuration="2.057350716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.054606323 +0000 UTC m=+1673.748088559" watchObservedRunningTime="2026-10-30 21:41:33.057350716 +0000 UTC m=+1673.750832952" Oct 30 21:41:33.194783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.188269 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:33.194783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.188392 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.194783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.188436 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:33.436353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.436311 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:33.825075 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.825037 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.825604 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:33.825105 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:41:34.189730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:34.186852 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.189730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:34.186918 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:34.823739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:34.822058 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.823739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:34.822122 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:41:35.191732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:35.189296 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.191732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:35.189387 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:35.822732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:35.819342 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.822732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:35.819404 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:41:36.187560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:36.187527 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.187879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:36.187849 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:36.820019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:36.819982 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.822035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:36.821997 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 21:41:37.187840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:37.187802 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.188135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:37.188103 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:38.126642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.126615 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:38.127723 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.127689 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:38.128347 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.128333 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:41:38.189727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.189131 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.189727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.189204 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:38.465288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.464059 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:38.766689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:38.766603 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:39.185712 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:39.185664 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.186104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:39.185765 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:41:40.191471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:40.191438 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:40.192030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:40.191997 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:40.192691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:40.192669 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:41:43.234456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:43.234418 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:44.772822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:44.771140 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:45.803602 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:41:45.803569 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:41:47.962586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:47.962537 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:52.755082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:52.755041 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:57.459371 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:57.459333 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.152876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:41:59.152846 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:02.241910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:02.241857 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:06.813739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:06.813679 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:07.027090 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:07.027034 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:08.515585 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:08.515546 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.430560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:11.430518 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:15.803755 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:42:15.803699 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:42:16.009600 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:16.005042 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:17.482084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.482049 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:42:17.486353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.486334 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.608731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.606456 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664151 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664206 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664271 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664415 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664464 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqn2\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664508 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664541 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664571 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.664730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.664656 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766086 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqn2\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766133 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766154 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766176 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766511 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766227 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766511 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766318 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766511 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766343 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766511 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766387 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766522 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766543 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.766973 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.766927 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.767312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.767285 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.767435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.767418 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.767665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.767636 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.769018 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.768992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.769159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.769136 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.828156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.828131 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqn2\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:17.828284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:17.828174 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:18.101029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:18.100926 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:18.678117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:18.677184 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:18.678117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:18.677276 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:18.678117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:18.677322 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:18.679491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:18.678752 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:42:19.287340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:19.285092 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" event={"ID":"99c9b5fd-8833-4836-829e-abbfaaeef9e4","Type":"ContainerStarted","Data":"1d6d4478cf8db831fd4afe81f356b1b9bbe3105b69aac814c14f01ba829ef67d"} Oct 30 21:42:19.287340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:19.285170 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" event={"ID":"99c9b5fd-8833-4836-829e-abbfaaeef9e4","Type":"ContainerStarted","Data":"293d22bc6ec7c13c4c9a8cfce9e0a5ec06599f91c87b7b6d96fd34d3c74e7c88"} Oct 30 21:42:19.469207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:19.469152 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podStartSLOduration=2.469138308 podStartE2EDuration="2.469138308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.465440523 +0000 UTC m=+1720.158922759" watchObservedRunningTime="2026-10-30 21:42:19.469138308 +0000 UTC m=+1720.162620609" Oct 30 21:42:20.102100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:20.102057 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:20.103582 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:20.103558 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:20.103654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:20.103609 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:21.100998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:21.100955 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:21.102092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:21.102062 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:21.102228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:21.102118 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:22.106676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:22.106637 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.115894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:22.115860 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:22.116170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:22.116138 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:23.102237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:23.102198 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:23.102453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:23.102282 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:24.102245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:24.102205 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:24.102776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:24.102271 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:25.101588 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:25.101544 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:42:25.101776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:25.101610 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 21:42:25.732054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:25.732013 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.105693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:26.105622 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:26.106039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:26.106016 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:26.106742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:26.106697 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:42:30.254175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:30.254129 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.899248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:34.899208 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:37.708408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:37.708360 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:39.176884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:39.176059 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:39.453863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:39.453779 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:39.677583 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:39.677543 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:41.225265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:41.225234 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.404125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:44.404090 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:45.802690 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:42:45.802650 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:42:48.877351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:48.877308 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.436207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:53.436166 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.101733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:42:58.097765 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:02.738088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:02.738036 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:07.378354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:07.378319 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:12.165898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:12.165866 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.884316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.884279 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 21:43:14.888595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.888576 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.946232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.943271 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 21:43:14.989931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.989901 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"default-dockercfg-w6f95\"" Oct 30 21:43:14.990073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.990000 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"distinct-2-w1-19dd4f-kserve-self-signed-certs\"" Oct 30 21:43:14.992813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.992794 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"openshift-service-ca.crt\"" Oct 30 21:43:14.992813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.992807 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"kube-root-ca.crt\"" Oct 30 21:43:14.997451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997430 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2mm\" (UniqueName: \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.997516 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997466 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.997610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997592 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.997652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997627 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.997807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:14.997870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:14.997837 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.026964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.026932 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.099347 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099302 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099380 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099486 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2mm\" (UniqueName: \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099516 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099622 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099661 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099760 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.099860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.099815 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.100225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.100205 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.101974 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.101950 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.102355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.102333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.162022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.161960 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2mm\" (UniqueName: \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\") pod \"distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.202237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.202205 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:15.580569 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:43:15.580536 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05b0c6cb_bd2f_49e6_ba66_dc39749770bc.slice/crio-d2076bc9808c67de9ec45bc65633097998aaaa495e053a8a66b6b3d060b438bb WatchSource:0}: Error finding container d2076bc9808c67de9ec45bc65633097998aaaa495e053a8a66b6b3d060b438bb: Status 404 returned error can't find the container with id d2076bc9808c67de9ec45bc65633097998aaaa495e053a8a66b6b3d060b438bb Oct 30 21:43:15.584370 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.584346 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 21:43:15.746000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.745971 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" event={"ID":"05b0c6cb-bd2f-49e6-ba66-dc39749770bc","Type":"ContainerStarted","Data":"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42"} Oct 30 21:43:15.746145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.746009 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" event={"ID":"05b0c6cb-bd2f-49e6-ba66-dc39749770bc","Type":"ContainerStarted","Data":"d2076bc9808c67de9ec45bc65633097998aaaa495e053a8a66b6b3d060b438bb"} Oct 30 21:43:15.802299 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:43:15.802259 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:43:15.885526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:15.885472 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podStartSLOduration=1.885459113 podStartE2EDuration="1.885459113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:15.88206624 +0000 UTC m=+1776.575548476" watchObservedRunningTime="2026-10-30 21:43:15.885459113 +0000 UTC m=+1776.578941348" Oct 30 21:43:16.203187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:16.203112 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:16.531411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:16.531252 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:21.230395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:21.230307 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:25.203288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:25.203249 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:25.220720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:25.220681 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:25.849264 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:25.849155 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:43:25.947122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:25.947088 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.433505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:30.433466 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:35.110769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:35.110684 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.448768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.448689 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.664171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.664102 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.675273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.675252 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:43:39.675748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.675693 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:43:39.688574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.688553 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:43:39.689022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.689003 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:43:39.693469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.693454 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:43:39.693938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:39.693922 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:43:44.280303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:44.280266 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.807036 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:43:45.807003 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:43:48.732747 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:48.731173 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.356584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:53.355830 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.935021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:57.934982 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.705836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.705792 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 21:43:58.710266 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.710246 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.770441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.770408 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 21:43:58.771489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771461 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.771638 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771551 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.771739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771668 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cng\" (UniqueName: \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.771739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771701 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.771861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771755 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.771861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.771779 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.815684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.815633 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-991335\"/\"kube-root-ca.crt\"" Oct 30 21:43:58.815684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.815670 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-991335\"/\"openshift-service-ca.crt\"" Oct 30 21:43:58.815878 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.815673 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-991335\"/\"default-dockercfg-9scmz\"" Oct 30 21:43:58.816970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.816954 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-991335\"/\"distinct-2-w5-991335-kserve-self-signed-certs\"" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.903848 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cng\" (UniqueName: \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904182 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904267 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904304 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904381 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904537 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.904980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.904660 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.906151 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.905593 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.906686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.906628 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.906686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.906677 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.917819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.917791 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.920242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.920215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:58.966661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:58.966597 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cng\" (UniqueName: \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\") pod \"distinct-2-w5-991335-kserve-98fbdfccc-gtknl\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:59.022540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:59.022513 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:43:59.400034 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:43:59.400002 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b01a127_e0e7_44c5_a3da_efe7971bfad1.slice/crio-85a643aff556b505d90e6109c5dcb6b24f7f9473e94f00e579d267ce4a615ae4 WatchSource:0}: Error finding container 85a643aff556b505d90e6109c5dcb6b24f7f9473e94f00e579d267ce4a615ae4: Status 404 returned error can't find the container with id 85a643aff556b505d90e6109c5dcb6b24f7f9473e94f00e579d267ce4a615ae4 Oct 30 21:43:59.403879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:43:59.403853 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 21:44:00.183406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:00.183365 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" event={"ID":"4b01a127-e0e7-44c5-a3da-efe7971bfad1","Type":"ContainerStarted","Data":"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c"} Oct 30 21:44:00.183861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:00.183418 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" event={"ID":"4b01a127-e0e7-44c5-a3da-efe7971bfad1","Type":"ContainerStarted","Data":"85a643aff556b505d90e6109c5dcb6b24f7f9473e94f00e579d267ce4a615ae4"} Oct 30 21:44:00.305769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:00.305700 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podStartSLOduration=2.305684393 podStartE2EDuration="2.305684393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:00.304316397 +0000 UTC m=+1820.997798632" watchObservedRunningTime="2026-10-30 21:44:00.305684393 +0000 UTC m=+1820.999166629" Oct 30 21:44:01.023366 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:01.023332 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:44:02.545658 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:02.545612 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.211540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.211503 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:44:06.216799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.216777 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.269782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.269748 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:44:06.270610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.270563 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"kube-root-ca.crt\"" Oct 30 21:44:06.270755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.270654 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"openshift-service-ca.crt\"" Oct 30 21:44:06.270937 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.270888 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"default-dockercfg-dlzxb\"" Oct 30 21:44:06.297995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.297970 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"distinct-2-w4-554f46-kserve-self-signed-certs\"" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351049 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351163 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm4m\" (UniqueName: \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351245 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351329 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351387 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.354538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.351547 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.376862 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.371986 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.453275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453240 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453343 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm4m\" (UniqueName: \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453459 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453509 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453860 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453884 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.453972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.453936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.455643 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.455621 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.455945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.455928 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.516046 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.515976 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm4m\" (UniqueName: \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\") pod \"distinct-2-w4-554f46-kserve-85667b8896-skm2x\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.531794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.531772 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:06.915697 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:44:06.915665 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode008f318_954f_498c_8fc2_e638bfa5bba9.slice/crio-af7fde26497c3ef26ea2a552872d0eed499f4bdb8d6d300bce769144f9e17973 WatchSource:0}: Error finding container af7fde26497c3ef26ea2a552872d0eed499f4bdb8d6d300bce769144f9e17973: Status 404 returned error can't find the container with id af7fde26497c3ef26ea2a552872d0eed499f4bdb8d6d300bce769144f9e17973 Oct 30 21:44:06.920550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:06.920525 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:44:07.240577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:07.240491 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" event={"ID":"e008f318-954f-498c-8fc2-e638bfa5bba9","Type":"ContainerStarted","Data":"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1"} Oct 30 21:44:07.240577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:07.240539 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" event={"ID":"e008f318-954f-498c-8fc2-e638bfa5bba9","Type":"ContainerStarted","Data":"af7fde26497c3ef26ea2a552872d0eed499f4bdb8d6d300bce769144f9e17973"} Oct 30 21:44:07.390205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:07.390145 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podStartSLOduration=1.390127194 podStartE2EDuration="1.390127194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:07.388678139 +0000 UTC m=+1828.082160376" watchObservedRunningTime="2026-10-30 21:44:07.390127194 +0000 UTC m=+1828.083609432" Oct 30 21:44:07.392299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:07.392261 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:07.532245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:07.532159 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:09.022947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:09.022915 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:44:09.035764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:09.035738 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:44:09.273507 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:09.271920 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:44:12.019527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:12.013764 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:12.823047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:12.822975 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:15.804241 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:44:15.804198 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:44:16.532261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:16.532227 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:16.545052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:16.545028 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:17.012570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:17.012528 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:17.338370 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:17.338279 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:44:18.763495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.763455 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 21:44:18.772109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.772088 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.823242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.823212 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 21:44:18.869637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.869604 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71aab5\"/\"kube-root-ca.crt\"" Oct 30 21:44:18.869637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.869620 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71aab5\"/\"openshift-service-ca.crt\"" Oct 30 21:44:18.874423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874392 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.874568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874429 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.874568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874509 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.874662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874644 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.874701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874675 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7btn\" (UniqueName: \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.874844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.874824 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.876455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.876435 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71aab5\"/\"default-dockercfg-25t4z\"" Oct 30 21:44:18.876744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.876728 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71aab5\"/\"distinct-2-w3-71aab5-kserve-self-signed-certs\"" Oct 30 21:44:18.910839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.910775 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:18.976256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976212 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976260 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976309 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976361 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976390 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7btn\" (UniqueName: \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976475 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.976959 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.976939 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.977620 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.977594 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.978256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.978225 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.980243 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.980218 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:18.980777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:18.980756 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:19.045474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:19.045403 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7btn\" (UniqueName: \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\") pod \"distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:19.094192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:19.094156 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:19.509313 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:44:19.509280 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b595f3_10e5_4df8_990c_f49770ba0924.slice/crio-47a81de3284da1e41e8d9e454aef2c4b6bdf1c1f750fd1428a18d6a7dc20fe79 WatchSource:0}: Error finding container 47a81de3284da1e41e8d9e454aef2c4b6bdf1c1f750fd1428a18d6a7dc20fe79: Status 404 returned error can't find the container with id 47a81de3284da1e41e8d9e454aef2c4b6bdf1c1f750fd1428a18d6a7dc20fe79 Oct 30 21:44:19.512195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:19.512171 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 21:44:20.352117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:20.352084 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" event={"ID":"89b595f3-10e5-4df8-990c-f49770ba0924","Type":"ContainerStarted","Data":"5621eb888415687f777bb964a0a0871fa5ebd7a2b259ee603f7118392c322314"} Oct 30 21:44:20.352518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:20.352129 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" event={"ID":"89b595f3-10e5-4df8-990c-f49770ba0924","Type":"ContainerStarted","Data":"47a81de3284da1e41e8d9e454aef2c4b6bdf1c1f750fd1428a18d6a7dc20fe79"} Oct 30 21:44:20.473862 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:20.473807 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podStartSLOduration=2.473793445 podStartE2EDuration="2.473793445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:20.473356922 +0000 UTC m=+1841.166839158" watchObservedRunningTime="2026-10-30 21:44:20.473793445 +0000 UTC m=+1841.167275680" Oct 30 21:44:21.094273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:21.094238 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:21.716287 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:21.716247 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.341506 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:26.341468 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.554187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:26.554147 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:29.095087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:29.095039 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:29.108483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:29.108450 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:29.474473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:29.474435 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:44:30.973103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:30.973061 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.930346 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:32.930314 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 21:44:32.941084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:32.941049 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.076882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.076841 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.077137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077106 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.077270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077179 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.077270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077249 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.077380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077297 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.077380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077346 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2v7\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.078436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077419 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.078436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077468 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.078436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.077490 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.084177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.084148 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 21:44:33.179446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.179737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179673 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.179874 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179768 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.179874 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179833 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179879 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.179929 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2v7\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.180022 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.180079 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.180104 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.180566 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.180504 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.181099 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.181075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.181586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.181561 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.181772 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.181695 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.183485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.183461 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.184438 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.184412 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.242132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.242101 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2v7\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.243189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.243158 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.256686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.256666 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:33.541431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.541366 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:44:33.546774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.546753 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.634875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.634839 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:44:33.685910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.685873 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686058 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686084 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf66\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686120 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686159 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686284 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686366 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686395 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.686488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.686479 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.717641 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:44:33.717599 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb15ad2_f865_4b84_b4ca_30b3925a134c.slice/crio-0bc849af7767de3fed4d3c0693928fd60a9feaf7e183965e1ac21e1e491938ea WatchSource:0}: Error finding container 0bc849af7767de3fed4d3c0693928fd60a9feaf7e183965e1ac21e1e491938ea: Status 404 returned error can't find the container with id 0bc849af7767de3fed4d3c0693928fd60a9feaf7e183965e1ac21e1e491938ea Oct 30 21:44:33.720492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.720453 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:33.720587 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.720535 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:33.720587 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.720574 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:33.722446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.722423 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 21:44:33.787474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787441 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787607 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787637 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf66\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787667 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787693 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787745 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.787820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787803 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.788032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787831 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.788032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.788032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.787914 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.788294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.788272 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.789602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.788915 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.789602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.789351 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.789602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.789559 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.790473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.790451 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.791233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.791209 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.853775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.853690 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.855077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.855048 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf66\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:33.861938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:33.861907 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:34.314749 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:44:34.311410 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad891f0b_3f80_471f_b1e2_5c53bf70fe09.slice/crio-9a25a0881422c2904776dac136043bcf51f22aa6e97e0ee0b93d7609d31f154a WatchSource:0}: Error finding container 9a25a0881422c2904776dac136043bcf51f22aa6e97e0ee0b93d7609d31f154a: Status 404 returned error can't find the container with id 9a25a0881422c2904776dac136043bcf51f22aa6e97e0ee0b93d7609d31f154a Oct 30 21:44:34.315313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.314986 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:34.315313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.315070 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:34.315313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.315109 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:34.315797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.315733 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:44:34.511447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.511336 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" event={"ID":"dcb15ad2-f865-4b84-b4ca-30b3925a134c","Type":"ContainerStarted","Data":"f70fc98ea244d819178e2587b9d3c86473c891269c507ab1d5eb4f7707ca9e10"} Oct 30 21:44:34.511447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.511410 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" event={"ID":"dcb15ad2-f865-4b84-b4ca-30b3925a134c","Type":"ContainerStarted","Data":"0bc849af7767de3fed4d3c0693928fd60a9feaf7e183965e1ac21e1e491938ea"} Oct 30 21:44:34.513384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.513353 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" event={"ID":"ad891f0b-3f80-471f-b1e2-5c53bf70fe09","Type":"ContainerStarted","Data":"f642ae2debb0053b4510f1fa96b8ae861a7bbabe4b352f98861870b7ac9d3b8d"} Oct 30 21:44:34.513528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.513419 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" event={"ID":"ad891f0b-3f80-471f-b1e2-5c53bf70fe09","Type":"ContainerStarted","Data":"9a25a0881422c2904776dac136043bcf51f22aa6e97e0ee0b93d7609d31f154a"} Oct 30 21:44:34.644752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.644063 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podStartSLOduration=2.644050033 podStartE2EDuration="2.644050033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:34.642629725 +0000 UTC m=+1855.336111986" watchObservedRunningTime="2026-10-30 21:44:34.644050033 +0000 UTC m=+1855.337532330" Oct 30 21:44:34.824838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.824681 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podStartSLOduration=1.8246647679999999 podStartE2EDuration="1.824664768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:34.800472752 +0000 UTC m=+1855.493954989" watchObservedRunningTime="2026-10-30 21:44:34.824664768 +0000 UTC m=+1855.518147004" Oct 30 21:44:34.862280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:34.862245 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:35.257162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.257113 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:35.258746 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.258720 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:35.258860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.258768 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:35.864035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.863836 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:35.864035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.863903 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:35.969229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:35.969169 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.257363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:36.257267 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:36.257363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:36.257330 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:36.862894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:36.862859 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:36.863076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:36.862926 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:37.257147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:37.257058 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:37.257147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:37.257120 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:37.862768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:37.862737 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:37.862941 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:37.862799 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:38.257128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:38.257033 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:38.257128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:38.257082 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:38.862336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:38.862303 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:38.862561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:38.862542 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:39.257103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.257013 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:39.257103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.257083 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:39.477761 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.477727 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:39.705375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.705337 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:39.863118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.863084 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:39.863286 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.863137 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:39.996784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:39.987529 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.257201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.257100 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:44:40.257201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.257150 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:44:40.743763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.743730 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.863079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.863039 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:44:40.863272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.863113 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:44:40.955441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:40.955401 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.549262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:41.549229 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:41.551136 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:41.550976 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:41.552164 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:41.552135 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 21:44:42.476755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:42.474992 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:42.477266 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:42.477140 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:42.484414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:42.479094 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:44:45.924134 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:44:45.922915 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:44:46.636251 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:46.636196 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:51.291142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:51.291104 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:55.984134 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:44:55.984097 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.244992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.244954 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:00.251348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.251324 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.324499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.324462 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.324753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.324729 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6znh\" (UniqueName: \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.350637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.350607 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:00.391061 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.391027 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:45:00.426782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.426741 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.426959 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.426940 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6znh\" (UniqueName: \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.427456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.427437 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.486422 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.486391 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6znh\" (UniqueName: \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.566011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.565924 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:00.677172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.677082 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.967354 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:00.967312 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79fe475_9a83_4fe2_a393_2bcbcf07fe78.slice/crio-010184ee5529a84708363575a7d6fd5d5af24fc4f0d0fa806271daace47db5e3 WatchSource:0}: Error finding container 010184ee5529a84708363575a7d6fd5d5af24fc4f0d0fa806271daace47db5e3: Status 404 returned error can't find the container with id 010184ee5529a84708363575a7d6fd5d5af24fc4f0d0fa806271daace47db5e3 Oct 30 21:45:00.970322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.969877 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:45:00.970637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:00.970606 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:01.723147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.723104 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:45:01.728293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.728268 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.777365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.777325 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:45:01.784988 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.784959 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.823780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.823747 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:45:01.860639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.860595 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.860842 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.860736 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.860842 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.860809 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.860979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.860846 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861037 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.860972 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slncp\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861064 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861087 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861131 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861160 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861185 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861217 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861231 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861276 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861328 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861360 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.861494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861444 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6pg\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861761 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861738 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.861892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.861778 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.865597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.865573 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:45:01.884560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.884520 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerStarted","Data":"010184ee5529a84708363575a7d6fd5d5af24fc4f0d0fa806271daace47db5e3"} Oct 30 21:45:01.963637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963605 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.963827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963645 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.963875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963837 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.963918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963873 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.963962 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963924 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963958 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.963995 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slncp\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964083 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964110 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964184 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964226 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964248 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964296 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964319 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964364 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964384 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964390 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964444 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964470 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.964655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964518 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6pg\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.965179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964774 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.965179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.964889 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.965179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965027 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.965179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965123 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.965471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965350 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.965471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965443 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.965750 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965464 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.965750 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.965676 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.966632 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.966607 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:01.967229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.967198 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.967435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.967415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:01.968848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:01.968829 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:02.022437 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.022374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:02.029537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.029359 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:02.030022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.030001 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slncp\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:02.030120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.030057 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6pg\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:02.045526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.045501 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:02.110116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.108446 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:02.488171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.488138 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:45:02.518001 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:02.517971 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163a23c1_1c65_40eb_a8c6_c63d2687aa93.slice/crio-9c8bc1e5457b5f294036e7756ed3221bece790bd52dab564a8790dceef46811e WatchSource:0}: Error finding container 9c8bc1e5457b5f294036e7756ed3221bece790bd52dab564a8790dceef46811e: Status 404 returned error can't find the container with id 9c8bc1e5457b5f294036e7756ed3221bece790bd52dab564a8790dceef46811e Oct 30 21:45:02.520125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.520088 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.520226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.520163 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.520226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.520195 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.532893 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:02.532868 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3009c681_1137_4f35_a754_9e535285bd45.slice/crio-b5a306a5e84df68ea4b46a8a94473ce17706a161a757aed30d42d5d1f8c17cd6 WatchSource:0}: Error finding container b5a306a5e84df68ea4b46a8a94473ce17706a161a757aed30d42d5d1f8c17cd6: Status 404 returned error can't find the container with id b5a306a5e84df68ea4b46a8a94473ce17706a161a757aed30d42d5d1f8c17cd6 Oct 30 21:45:02.534519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.534498 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:45:02.536490 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.536460 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.536576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.536522 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.536576 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.536548 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:02.898074 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.898037 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" event={"ID":"3009c681-1137-4f35-a754-9e535285bd45","Type":"ContainerStarted","Data":"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004"} Oct 30 21:45:02.898584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.898100 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" event={"ID":"3009c681-1137-4f35-a754-9e535285bd45","Type":"ContainerStarted","Data":"b5a306a5e84df68ea4b46a8a94473ce17706a161a757aed30d42d5d1f8c17cd6"} Oct 30 21:45:02.900605 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.900571 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" event={"ID":"163a23c1-1c65-40eb-a8c6-c63d2687aa93","Type":"ContainerStarted","Data":"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a"} Oct 30 21:45:02.900760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:02.900631 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" event={"ID":"163a23c1-1c65-40eb-a8c6-c63d2687aa93","Type":"ContainerStarted","Data":"9c8bc1e5457b5f294036e7756ed3221bece790bd52dab564a8790dceef46811e"} Oct 30 21:45:03.027727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.027666 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podStartSLOduration=2.027651341 podStartE2EDuration="2.027651341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.025812611 +0000 UTC m=+1883.719294859" watchObservedRunningTime="2026-10-30 21:45:03.027651341 +0000 UTC m=+1883.721133577" Oct 30 21:45:03.046358 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.046305 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:03.047849 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.047810 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:03.047985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.047872 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:03.112835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.111545 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:03.113081 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.113055 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:03.113139 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.113117 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:03.145623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:03.145566 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podStartSLOduration=2.14555123 podStartE2EDuration="2.14555123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.142967403 +0000 UTC m=+1883.836449638" watchObservedRunningTime="2026-10-30 21:45:03.14555123 +0000 UTC m=+1883.839033465" Oct 30 21:45:04.046250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:04.046211 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:04.046647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:04.046274 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:04.126810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:04.120833 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:04.126810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:04.120898 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:05.046560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:05.046182 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:05.046560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:05.046247 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:05.130786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:05.129339 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:05.130786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:05.129403 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:05.830834 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:05.828941 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:06.046176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:06.046142 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:06.046375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:06.046200 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:06.111786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:06.111101 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:06.111786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:06.111158 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:07.046491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:07.046453 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:07.046786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:07.046756 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:07.118024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:07.117907 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:07.118024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:07.117967 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:08.046992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.046965 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:08.047165 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.047145 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:08.112098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.112061 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:08.112365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.112335 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:08.988965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.988929 4918 generic.go:356] "Generic (PLEG): container finished" podID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerID="c07de4d9536e31064be1a6b1522fc24a6c2f38b3bb8abc216981c70ab8b6d057" exitCode=60 Oct 30 21:45:08.989525 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.989022 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerDied","Data":"c07de4d9536e31064be1a6b1522fc24a6c2f38b3bb8abc216981c70ab8b6d057"} Oct 30 21:45:08.989525 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:08.989323 4918 scope.go:122] "RemoveContainer" containerID="c07de4d9536e31064be1a6b1522fc24a6c2f38b3bb8abc216981c70ab8b6d057" Oct 30 21:45:09.046940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:09.046872 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:45:09.047646 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:09.047284 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:45:09.109785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:09.109759 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:45:09.109954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:09.109807 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:45:10.004273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.004237 4918 generic.go:356] "Generic (PLEG): container finished" podID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerID="089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c" exitCode=60 Oct 30 21:45:10.005023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.004404 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerDied","Data":"089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c"} Oct 30 21:45:10.005023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.004467 4918 scope.go:122] "RemoveContainer" containerID="c07de4d9536e31064be1a6b1522fc24a6c2f38b3bb8abc216981c70ab8b6d057" Oct 30 21:45:10.006037 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.006008 4918 scope.go:122] "RemoveContainer" containerID="089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c" Oct 30 21:45:10.006624 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:10.006454 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx_odh-ai-gateway-infra(d79fe475-9a83-4fe2-a393-2bcbcf07fe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" Oct 30 21:45:10.066497 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.066464 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:10.067670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.066824 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:10.068589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.068567 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:45:10.113302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.113281 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:10.113572 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.113545 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:10.114272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.114255 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:45:10.801495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:10.801459 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:11.014914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:11.014874 4918 scope.go:122] "RemoveContainer" containerID="089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c" Oct 30 21:45:11.015321 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:11.015123 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx_odh-ai-gateway-infra(d79fe475-9a83-4fe2-a393-2bcbcf07fe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" Oct 30 21:45:15.618788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:15.618752 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:15.805959 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:15.805896 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:45:17.294014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.293973 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:45:17.299914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.299890 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.381936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.381876 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvxf\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.389838 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.389922 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.389968 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.390012 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.390113 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.390170 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.390267 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.390371 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.391881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.391142 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:45:17.491404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491365 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491432 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491510 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491565 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491652 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491757 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491865 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvxf\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.491918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491897 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.492030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.491952 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.492107 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.492077 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.492412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.492385 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.493340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.493316 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.493935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.493691 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.493935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.493884 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.498748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.495994 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.498748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.496351 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.550759 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.550651 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.554221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.554176 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvxf\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.615788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.615737 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:17.655725 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.655676 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:45:17.661555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.661526 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.696009 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.695975 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:45:17.706348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.702701 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.740736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.740654 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:45:17.759343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.759302 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798620 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798758 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798788 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798824 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxxc\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798872 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798925 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.798966 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799330 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799380 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4tk\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799421 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799564 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799642 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799677 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799730 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799763 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799800 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799839 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.800835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.799875 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.901157 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.901114 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.901348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.901189 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.901348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.901258 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.901348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.901300 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4tk\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.901591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.901374 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.902103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902052 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.902227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902133 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.902288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902252 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.902378 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902347 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.902482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.902818 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902797 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.902916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902838 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.902916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902893 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.903030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.902924 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.903277 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903256 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.903277 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903272 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.903417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903193 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.903417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903303 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.903417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903338 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.903417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.903374 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.905736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.904982 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.905736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.905400 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.905893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.905833 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.905893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.905875 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.905995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.905973 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.906433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.906416 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.906572 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.906557 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxxc\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.906731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.906695 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.906981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.906957 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.907049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.906374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.907049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.907023 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.907390 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.907355 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.959895 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.959867 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4tk\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.961543 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.961517 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.961825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.961802 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:17.964640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.964618 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxxc\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:17.982452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:17.982431 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:18.021155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.021127 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:45:18.021319 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:18.021280 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce138bce_f56c_4abb_9b2e_efbdb3b342ed.slice/crio-51074560a2d40ff5d05cd82ccd5db33a7748c7b499a049ea465bfd44b640fec4 WatchSource:0}: Error finding container 51074560a2d40ff5d05cd82ccd5db33a7748c7b499a049ea465bfd44b640fec4: Status 404 returned error can't find the container with id 51074560a2d40ff5d05cd82ccd5db33a7748c7b499a049ea465bfd44b640fec4 Oct 30 21:45:18.023834 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.023785 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.023935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.023881 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.023935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.023927 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.026065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.026035 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:18.076612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.076422 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" event={"ID":"ce138bce-f56c-4abb-9b2e-efbdb3b342ed","Type":"ContainerStarted","Data":"51074560a2d40ff5d05cd82ccd5db33a7748c7b499a049ea465bfd44b640fec4"} Oct 30 21:45:18.372539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.372513 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:45:18.384679 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.384635 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.384842 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.384755 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.384842 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.384802 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.487878 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.486201 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:45:18.492017 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.491978 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.492127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.492067 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.492127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.492112 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:18.499293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.499271 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:45:18.507947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.506686 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.585763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.585135 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:45:18.625109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.625081 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.625420 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.625398 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.625599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.625575 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.626228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.626206 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.626588 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.626472 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.626835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.626799 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dts\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.627269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.627218 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.627894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.627873 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.629062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.628140 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730591 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730631 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730656 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730742 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730760 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730791 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dts\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.730909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730813 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.731213 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730923 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.731213 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.730955 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.732610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.731835 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.732610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.732136 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.732610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.732202 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.732610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.732273 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.732920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.732700 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.734467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.734445 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.734573 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.734511 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.790838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.790806 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dts\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.791355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.791329 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:18.895951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:18.895855 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:19.130683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.130643 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" event={"ID":"ce138bce-f56c-4abb-9b2e-efbdb3b342ed","Type":"ContainerStarted","Data":"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb"} Oct 30 21:45:19.134334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.134253 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" event={"ID":"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c","Type":"ContainerStarted","Data":"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327"} Oct 30 21:45:19.134564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.134545 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" event={"ID":"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c","Type":"ContainerStarted","Data":"23ccb12420ef5384c594ca31416de30cb00c0e84b5a4f928ba792dc7d20374cc"} Oct 30 21:45:19.138549 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.138525 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" event={"ID":"5db7213c-a8ca-48af-9a72-fe35e52e4ed8","Type":"ContainerStarted","Data":"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee"} Oct 30 21:45:19.138724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.138693 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" event={"ID":"5db7213c-a8ca-48af-9a72-fe35e52e4ed8","Type":"ContainerStarted","Data":"cab77f6acc02bd68489f6a2f2e6fc61f1cff5796829faea5b4e0bcd0bd93aa1f"} Oct 30 21:45:19.263325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.263167 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podStartSLOduration=2.263147324 podStartE2EDuration="2.263147324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.256542015 +0000 UTC m=+1899.950024251" watchObservedRunningTime="2026-10-30 21:45:19.263147324 +0000 UTC m=+1899.956629561" Oct 30 21:45:19.630869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.630829 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.634992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.634946 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:19.718732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.715083 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:19.753892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.753846 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:19.754149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.754129 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:19.754742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.721215 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podStartSLOduration=2.721191256 podStartE2EDuration="2.721191256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.440342954 +0000 UTC m=+1900.133825191" watchObservedRunningTime="2026-10-30 21:45:19.721191256 +0000 UTC m=+1900.414673493" Oct 30 21:45:19.986784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.986187 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.986784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:19.986248 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:20.085879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.085839 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.086160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.086126 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:20.201729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.100382 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podStartSLOduration=3.100362002 podStartE2EDuration="3.100362002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.563305714 +0000 UTC m=+1900.256787992" watchObservedRunningTime="2026-10-30 21:45:20.100362002 +0000 UTC m=+1900.793844242" Oct 30 21:45:20.279786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.262836 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:20.279786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.262902 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:45:20.279786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.262941 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:20.279786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.262966 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:20.394729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.394391 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" event={"ID":"241d1ba1-dcc2-46df-a016-6d021f131514","Type":"ContainerStarted","Data":"12160fd7ebc5de5aacf0ef917658fca0eb75b2f1e746cff7fe8b28ee20609f53"} Oct 30 21:45:20.651742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.624585 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.651742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.637420 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:21.026731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.997059 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.026731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:20.997118 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:21.064731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.040797 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.064731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.040865 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:21.064731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.059589 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:21.531438 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.531396 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" event={"ID":"241d1ba1-dcc2-46df-a016-6d021f131514","Type":"ContainerStarted","Data":"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794"} Oct 30 21:45:21.633131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.633095 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.633405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.633358 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:21.674733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.667491 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podStartSLOduration=3.667470668 podStartE2EDuration="3.667470668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:21.629557288 +0000 UTC m=+1902.323039525" watchObservedRunningTime="2026-10-30 21:45:21.667470668 +0000 UTC m=+1902.360952905" Oct 30 21:45:21.903839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.903795 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:21.915740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.905401 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.915740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.905458 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:21.915740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.907959 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:21.993647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.993441 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.994149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:21.993941 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:22.027616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.027578 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.027902 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.027864 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:22.281929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.269869 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.376292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.376243 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.633158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.633107 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.633851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.633402 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:22.900468 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.900382 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.901055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.901021 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:22.915731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.907362 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.001670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.935192 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:45:23.006092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:22.998466 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.006323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.006292 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:23.006521 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.006503 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:45:23.051117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.048823 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.051117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.048877 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:23.296657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.296554 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.304737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.297255 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:45:23.405739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.398221 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:23.405739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.398909 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:45:23.489773 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.486797 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:45:23.639738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.638957 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.643897 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.643861 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:23.742259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:23.742213 4918 scope.go:122] "RemoveContainer" containerID="089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c" Oct 30 21:45:24.039786 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.036098 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.111078 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.101167 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.111156 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.111029 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.111527 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:24.125385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.064586 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.368875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.368829 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.437267 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.390006 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:45:24.442876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.442847 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:45:24.458654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.403946 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:24.567838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.563233 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.593162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.593132 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:45:24.690779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.682064 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.690779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:24.682142 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:25.067755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.058506 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.067755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.058580 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:25.067755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.058845 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.145844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.133159 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.145844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.133240 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:25.145844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.133786 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.145844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.133854 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:25.162748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.162204 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.446360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.446304 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.447062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.447005 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:25.637765 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.635291 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:45:25.688733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.639937 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.813014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.812961 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.813337 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:25.813304 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:26.100240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.075598 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.100240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.075672 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:26.154561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.154524 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.154850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.154820 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.175861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.175823 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.176126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.176096 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:26.176344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.176324 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.176477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.176452 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:26.227939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.227886 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.228257 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.228224 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.254731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.230002 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.403255 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.403214 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.403548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.403517 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.446873 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.446818 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.477239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.449184 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.477239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.471903 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.477239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.471989 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.582176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.531887 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.582176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.531965 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.761785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.685039 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.820764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.810313 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.883737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.836603 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.883737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.836665 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:26.925143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.925101 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.925387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:26.925355 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:26.954734 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:26.939741 4918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.482s" Oct 30 21:45:27.001542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.001503 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.001867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.001835 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:27.153841 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.043056 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.176836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.176772 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:27.178219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.130149 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.178411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.130053 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:27.178529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.178507 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.197687 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.170911 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:27.251733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.202101 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.345732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.315169 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.448666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.448609 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.450342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.450301 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.547732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.543821 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 30 21:45:27.625889 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.625836 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:27.626189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.626143 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.626361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.626333 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.626445 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.626384 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:27.691298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.691193 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerStarted","Data":"36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f"} Oct 30 21:45:27.692837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.692757 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.823762 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.819154 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.847908 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.844076 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" podStartSLOduration=21.319263371 podStartE2EDuration="27.844053522s" totalImagesPullingTime="6.524790151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:27.814049217 +0000 UTC m=+1908.507531481" watchObservedRunningTime="2026-10-30 21:45:27.844053522 +0000 UTC m=+1908.537535760" Oct 30 21:45:27.937731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.934332 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.937731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:27.934390 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:28.031888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.031851 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.032757 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.032675 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:28.032981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.032112 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.033112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.033092 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:28.335801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.324582 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:45:28.514734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.468044 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.514734 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.451633 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.548687 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.548618 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:28.574990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.574907 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.724777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.716832 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.724777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.716922 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:28.724777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.713941 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.927733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.883655 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:28.956016 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.938456 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.061334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.061232 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:29.069847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.036771 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:29.070150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:28.985035 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.070347 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.070312 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:29.090102 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.090067 4918 generic.go:356] "Generic (PLEG): container finished" podID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerID="36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f" exitCode=60 Oct 30 21:45:29.090578 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.090546 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerDied","Data":"36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f"} Oct 30 21:45:29.090811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.090789 4918 scope.go:122] "RemoveContainer" containerID="089089ab9154362ebd705de4baf699eb64fa661f0fee32f85a9f3d614309956c" Oct 30 21:45:29.091118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.091091 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.094833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.094778 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:29.424785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.402460 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:45:29.480925 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.480879 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:29.596758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.580729 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:29.621147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.621112 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.621423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.621393 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:29.759823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:29.734827 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:45:30.058984 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.058872 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:30.074724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.066792 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 30 21:45:30.074724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.067230 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.074724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.067297 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:30.074724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.067399 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.074724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.067433 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:30.160728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.098167 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.160728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.098223 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:30.160728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.120654 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:30.550733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.504902 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:30.674024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.673976 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:30.689270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.689217 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:45:30.842855 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.695626 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.843168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.843118 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:30.843319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:30.800355 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:45:31.097125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.097006 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:31.218995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.218135 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:31.218995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.218217 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:31.272467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.272431 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:31.272726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.272683 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:31.370730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.369208 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:31.370730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.369267 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:31.449733 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:31.444437 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79fe475_9a83_4fe2_a393_2bcbcf07fe78.slice/crio-36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79fe475_9a83_4fe2_a393_2bcbcf07fe78.slice/crio-conmon-36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:31.547754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.534424 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:31.624662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.624574 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:31.642956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.642911 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:31.818169 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.792881 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.012691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:31.881389 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:32.113093 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.113042 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:45:32.114128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.011174 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:32.136475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.136422 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:32.137488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.114545 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:32.169986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.169886 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:32.192880 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.170231 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:32.210125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.210082 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:45:32.420790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.371597 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:32.420790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.420150 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:32.666737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.664337 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:32.666737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.665422 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:32.719798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.699863 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:45:32.719798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.699944 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:45:32.791982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.791933 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:32.925321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:32.925286 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:33.048949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.048861 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:33.050135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.050005 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:33.050313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.050290 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:33.053885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.053861 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:33.074750 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.068301 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:33.077348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.077323 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\") pod \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " Oct 30 21:45:33.077856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.077828 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\" (UniqueName: \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\") pod \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\" (UID: \"d79fe475-9a83-4fe2-a393-2bcbcf07fe78\") " Oct 30 21:45:33.139236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.133474 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:33.139236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.137788 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:33.158851 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.158816 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca" pod "d79fe475-9a83-4fe2-a393-2bcbcf07fe78" (UID: "d79fe475-9a83-4fe2-a393-2bcbcf07fe78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:33.168847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.162273 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:45:33.168847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.162287 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:33.182850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.170298 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" event={"ID":"d79fe475-9a83-4fe2-a393-2bcbcf07fe78","Type":"ContainerDied","Data":"010184ee5529a84708363575a7d6fd5d5af24fc4f0d0fa806271daace47db5e3"} Oct 30 21:45:33.223477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.194700 4918 scope.go:122] "RemoveContainer" containerID="36f0819d43039f102f9032697d02b7e49effbd89c8e5514ee7b092a2faeb511f" Oct 30 21:45:33.223477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.193114 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx" Oct 30 21:45:33.223477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.196928 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:33.223477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.196980 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:33.223477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.199288 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:45:33.263840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.263810 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-maas-api-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.269591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.269399 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh" pod "d79fe475-9a83-4fe2-a393-2bcbcf07fe78" (UID: "d79fe475-9a83-4fe2-a393-2bcbcf07fe78"). InnerVolumeSpecName "kube-api-access-p6znh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:33.349519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.349434 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:45:33.402488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:33.402450 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6znh\" (UniqueName: \"kubernetes.io/projected/d79fe475-9a83-4fe2-a393-2bcbcf07fe78-kube-api-access-p6znh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.062977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.062898 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.093729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.066780 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:34.144005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.142086 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.144260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.144231 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:45:34.144380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.143907 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.144473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.144456 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 21:45:34.248403 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.248376 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:45:34.248751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.248694 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:45:34.248883 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.248872 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:34.462657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.461673 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:34.606980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.606944 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623825 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623857 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623893 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623901 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623930 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.623937 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.624126 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.628728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.624150 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:34.648802 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.630167 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.646235 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728280 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728495 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728532 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728574 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728669 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728792 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728825 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.728879 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxt7\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.757811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.661626 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:45:34.855413 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.838242 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.856208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.856181 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.856644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.856608 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.856823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.856806 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.857140 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.857109 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.869270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.857355 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.869270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.865142 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:34.869270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.865190 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.869682 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.869754 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.869806 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxt7\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.877517 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.877829 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.883246 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.878088 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.920822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.916244 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.927297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.927267 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.961782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.927787 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:34.962067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.932965 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 21:45:34.962237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.958568 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.962374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.962345 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:34.997126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.979114 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:34.999966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:34.999931 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:45:35.053344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.052903 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:35.146744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.121301 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxt7\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:35.204877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.204016 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:45:35.244047 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:35.244005 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79fe475_9a83_4fe2_a393_2bcbcf07fe78.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:35.248890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.244346 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:35.302779 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:35.269372 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79fe475_9a83_4fe2_a393_2bcbcf07fe78.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:35.332668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.332556 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:35.338839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.338810 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889945-k9mnx"] Oct 30 21:45:35.342297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.342273 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:35.351889 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.351849 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:45:35.435867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.435837 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" path="/var/lib/kubelet/pods/d79fe475-9a83-4fe2-a393-2bcbcf07fe78/volumes" Oct 30 21:45:35.458424 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.458379 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:35.503803 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.503775 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:35.511824 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.511798 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:45:35.916729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.915906 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:35.916729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.915965 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 21:45:35.943434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.943409 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:45:35.952977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.951411 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:35.952977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.951726 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:35.952977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:35.951770 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:36.413731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:36.413420 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" event={"ID":"2db2f944-3425-4d63-be41-4252daa4b028","Type":"ContainerStarted","Data":"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc"} Oct 30 21:45:36.413731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:36.413483 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" event={"ID":"2db2f944-3425-4d63-be41-4252daa4b028","Type":"ContainerStarted","Data":"e3d71158d6144b1818ab25339db065fd99a8bd3d75152ded13d95d359c9857f3"} Oct 30 21:45:36.460845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:36.460812 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:36.584769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:36.583170 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podStartSLOduration=3.583150796 podStartE2EDuration="3.583150796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:36.534590085 +0000 UTC m=+1917.228072324" watchObservedRunningTime="2026-10-30 21:45:36.583150796 +0000 UTC m=+1917.276633032" Oct 30 21:45:37.094730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:37.094147 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:37.458357 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:37.458106 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:37.460616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:37.460587 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:37.460865 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:37.460829 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:37.462865 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:37.462842 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:45:38.459263 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:38.459232 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:38.459647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:38.459287 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:39.459032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:39.458988 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:39.459249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:39.459045 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:40.167245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:40.167211 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:40.404393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:40.404355 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:40.459023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:40.458939 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:40.459023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:40.458995 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:41.459623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:41.459579 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:41.460078 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:41.459722 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:41.883728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:41.881999 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:42.459552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:42.459514 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:42.459833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:42.459640 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:43.459532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:43.459494 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:43.459784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:43.459559 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:45:44.463498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:44.463467 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:44.464026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:44.463843 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:44.464419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:44.464400 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:45:45.808006 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:45:45.807967 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:45:46.698159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:46.698119 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:49.134764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.134723 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:45:49.135544 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.135528 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79fe475-9a83-4fe2-a393-2bcbcf07fe78" containerName="cleanup" Oct 30 21:45:49.143487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.143467 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.198773 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.198741 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:45:49.252740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.252694 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.252924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.252751 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.252924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.252893 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.252957 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.252985 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.253003 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.253026 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlw7\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.253145 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.253275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.253180 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354522 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354566 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354588 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlw7\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354662 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354741 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.354918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354913 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355033 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354941 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355033 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.354912 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355037 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355054 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355096 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355131 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355252 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.355352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.355336 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.357319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.357295 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.357418 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.357342 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.413519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.413462 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.413718 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.413687 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlw7\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.458293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.458265 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:49.891100 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:49.891064 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f02365_e1e7_48ac_85c2_b39f0dca13f2.slice/crio-f0332a085d980d036f028f03280bc785e17f342d3b170621eb41b3c0fdb36a51 WatchSource:0}: Error finding container f0332a085d980d036f028f03280bc785e17f342d3b170621eb41b3c0fdb36a51: Status 404 returned error can't find the container with id f0332a085d980d036f028f03280bc785e17f342d3b170621eb41b3c0fdb36a51 Oct 30 21:45:49.891454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.891424 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:45:49.893943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.893907 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:49.894031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.893974 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:49.894031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:49.894003 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:50.550070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:50.550031 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" event={"ID":"55f02365-e1e7-48ac-85c2-b39f0dca13f2","Type":"ContainerStarted","Data":"6fa42ad8fefc5d49c89519004dbcf8f4696c4805a2cf507a40fb22cf256ca68a"} Oct 30 21:45:50.550070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:50.550086 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" event={"ID":"55f02365-e1e7-48ac-85c2-b39f0dca13f2","Type":"ContainerStarted","Data":"f0332a085d980d036f028f03280bc785e17f342d3b170621eb41b3c0fdb36a51"} Oct 30 21:45:50.763373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:50.763313 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podStartSLOduration=1.7632992920000001 podStartE2EDuration="1.763299292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:50.714644181 +0000 UTC m=+1931.408126416" watchObservedRunningTime="2026-10-30 21:45:50.763299292 +0000 UTC m=+1931.456781528" Oct 30 21:45:51.459227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:51.459192 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:51.460539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:51.460512 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:51.460611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:51.460571 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:51.538812 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:51.538759 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:52.459500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:52.459184 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:52.459500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:52.459232 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:53.458911 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:53.458864 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:53.459112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:53.458943 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:54.459318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:54.459284 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:54.459739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:54.459334 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:55.459179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:55.459147 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:55.459421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:55.459395 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:56.459171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:56.459137 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:45:56.459355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:56.459194 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:45:56.638158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:56.638120 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:57.463565 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:57.463536 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:57.463963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:57.463947 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:57.464539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:57.464518 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:45:58.177896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.177864 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11382-3c5c1a"] Oct 30 21:45:58.183797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.183765 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:45:58.232797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.232772 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11382-3c5c1a"] Oct 30 21:45:58.263608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.263583 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sxztf\"" Oct 30 21:45:58.268808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.268779 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkpq\" (UniqueName: \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\") pod \"test-curl-11382-3c5c1a\" (UID: \"1dd3742b-be57-4a7c-9212-c3024f6e2ce2\") " pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:45:58.369850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.369816 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkpq\" (UniqueName: \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\") pod \"test-curl-11382-3c5c1a\" (UID: \"1dd3742b-be57-4a7c-9212-c3024f6e2ce2\") " pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:45:58.394753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.394722 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11385-d18b3b"] Oct 30 21:45:58.399149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.399132 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:45:58.437058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.437001 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkpq\" (UniqueName: \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\") pod \"test-curl-11382-3c5c1a\" (UID: \"1dd3742b-be57-4a7c-9212-c3024f6e2ce2\") " pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:45:58.451127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.451104 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11385-d18b3b"] Oct 30 21:45:58.472354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.472330 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcxh\" (UniqueName: \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\") pod \"test-curl-11385-d18b3b\" (UID: \"de063a53-ee2e-48a6-a9bb-41a8eebbb126\") " pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:45:58.498021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.497982 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:45:58.574317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.574284 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcxh\" (UniqueName: \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\") pod \"test-curl-11385-d18b3b\" (UID: \"de063a53-ee2e-48a6-a9bb-41a8eebbb126\") " pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:45:58.633233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.633203 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcxh\" (UniqueName: \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\") pod \"test-curl-11385-d18b3b\" (UID: \"de063a53-ee2e-48a6-a9bb-41a8eebbb126\") " pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:45:58.712159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.712088 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:45:58.886119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:58.886080 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11382-3c5c1a"] Oct 30 21:45:58.886235 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:58.886148 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd3742b_be57_4a7c_9212_c3024f6e2ce2.slice/crio-19279dd50911b957857254ec034ab7e53fee7a4cf0e17cb2ebeac358eaa03fec WatchSource:0}: Error finding container 19279dd50911b957857254ec034ab7e53fee7a4cf0e17cb2ebeac358eaa03fec: Status 404 returned error can't find the container with id 19279dd50911b957857254ec034ab7e53fee7a4cf0e17cb2ebeac358eaa03fec Oct 30 21:45:59.097522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:59.097488 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11385-d18b3b"] Oct 30 21:45:59.099826 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:45:59.099800 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde063a53_ee2e_48a6_a9bb_41a8eebbb126.slice/crio-9f5bfef3bdf1a64dc37f2e0175ae24f869e472331c6851e9e007b3841af8bc8b WatchSource:0}: Error finding container 9f5bfef3bdf1a64dc37f2e0175ae24f869e472331c6851e9e007b3841af8bc8b: Status 404 returned error can't find the container with id 9f5bfef3bdf1a64dc37f2e0175ae24f869e472331c6851e9e007b3841af8bc8b Oct 30 21:45:59.671481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:59.671444 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11385-d18b3b" event={"ID":"de063a53-ee2e-48a6-a9bb-41a8eebbb126","Type":"ContainerStarted","Data":"9f5bfef3bdf1a64dc37f2e0175ae24f869e472331c6851e9e007b3841af8bc8b"} Oct 30 21:45:59.673113 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:45:59.673086 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11382-3c5c1a" event={"ID":"1dd3742b-be57-4a7c-9212-c3024f6e2ce2","Type":"ContainerStarted","Data":"19279dd50911b957857254ec034ab7e53fee7a4cf0e17cb2ebeac358eaa03fec"} Oct 30 21:46:00.683284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:00.683245 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11385-d18b3b" event={"ID":"de063a53-ee2e-48a6-a9bb-41a8eebbb126","Type":"ContainerStarted","Data":"238b044c4c3bd07fcbc76e88216a9f4278a7395f90bf38561b18d345dfa8b3a2"} Oct 30 21:46:00.685267 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:00.685229 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11382-3c5c1a" event={"ID":"1dd3742b-be57-4a7c-9212-c3024f6e2ce2","Type":"ContainerStarted","Data":"aff17963b84ecae7b9f4ff87c0421410996e53daee668524e51e3482295f88a1"} Oct 30 21:46:00.796354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:00.796293 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11385-d18b3b" podStartSLOduration=1.548412678 podStartE2EDuration="2.796274019s" totalImagesPullingTime="1.247861341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.790377973 +0000 UTC m=+1941.483860235" watchObservedRunningTime="2026-10-30 21:46:00.796274019 +0000 UTC m=+1941.489756256" Oct 30 21:46:00.910129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:00.910064 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11382-3c5c1a" podStartSLOduration=1.449106307 podStartE2EDuration="2.910047578s" totalImagesPullingTime="1.460941271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.905433058 +0000 UTC m=+1941.598915293" watchObservedRunningTime="2026-10-30 21:46:00.910047578 +0000 UTC m=+1941.603529815" Oct 30 21:46:01.322423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:01.322387 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:02.339023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.337482 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11385-d18b3b"] Oct 30 21:46:02.345011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.344979 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11385-d18b3b"] Oct 30 21:46:02.606961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.606870 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:46:02.661656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.660568 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:46:02.661656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.660851 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" containerID="cri-o://3202651ea4e6f2c0647c9e18fb7209852b7bff5e7a7fb6ab267b39954b073b6c" gracePeriod=30 Oct 30 21:46:02.661656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.661236 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" containerID="cri-o://fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f" gracePeriod=30 Oct 30 21:46:02.675295 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.675267 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11382-3c5c1a"] Oct 30 21:46:02.683267 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.683136 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11382-3c5c1a"] Oct 30 21:46:02.734417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.734374 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11382-3c5c1a" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" containerName="test-curl-11382-3c5c1a" containerID="cri-o://aff17963b84ecae7b9f4ff87c0421410996e53daee668524e51e3482295f88a1" gracePeriod=2 Oct 30 21:46:02.734919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:02.734644 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11385-d18b3b" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" containerName="test-curl-11385-d18b3b" containerID="cri-o://238b044c4c3bd07fcbc76e88216a9f4278a7395f90bf38561b18d345dfa8b3a2" gracePeriod=2 Oct 30 21:46:03.055738 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:03.055053 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11382-3c5c1a\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:46:03.121733 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:03.120959 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11385-d18b3b\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:46:03.524452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:03.524194 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:04.459082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:04.459034 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:46:04.485065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:04.485017 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.500322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:04.500276 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.699862 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:04.699810 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:46:04.857728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:04.856791 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.249747 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.222967 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.481902 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.481852 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.592787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.507821 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.592787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.554836 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:05.592787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.584038 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:05.749738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.744813 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.848776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:05.830295 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:06.074732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.057840 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:06.355827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.335081 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.468764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.445767 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:46:06.768732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.596896 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.770310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.770220 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.851975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.598184 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:46:06.869123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.693518 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:06.869409 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.869354 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.870021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:06.598151 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.073299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.073259 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:07.073606 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.073569 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.216785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.191861 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:07.251690 4918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.799s" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.290900 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.291032 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.291073 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.291109 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:46:07.323731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.291141 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:46:07.416733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.383115 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.573280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.573184 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.756776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.746917 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.840571 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.807100 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.883729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.880096 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.893823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:07.827757 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:46:08.093783 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.075828 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:08.358516 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.358422 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:46:08.759088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.758569 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.866252 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:08.866215 4918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.292s" Oct 30 21:46:08.866911 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.866890 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:46:08.900966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.900922 4918 cpu_manager.go:414] "containerMap: removing container" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" containerName="test-curl-11385-d18b3b" Oct 30 21:46:08.901201 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.901175 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" containerName="test-curl-11385-d18b3b" Oct 30 21:46:08.901407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.901391 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" containerName="test-curl-11382-3c5c1a" Oct 30 21:46:08.901530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.901513 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" containerName="test-curl-11382-3c5c1a" Oct 30 21:46:08.901948 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.901927 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" containerName="test-curl-11382-3c5c1a" Oct 30 21:46:08.902084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.902071 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" containerName="test-curl-11385-d18b3b" Oct 30 21:46:08.941943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.941910 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:46:08.957730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.943228 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:08.992810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.979390 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:08.992810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.979610 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:46:08.992810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:08.980274 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:46:09.119730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.118868 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a84c1592-openshift-default-dockercfg-2pppl\"" Oct 30 21:46:09.395730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.395439 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.725513 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.725426 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.725887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.725857 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.726163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.726142 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.726360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.726341 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.726556 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.726537 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.726774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.726754 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.726968 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.726949 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wt2t\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.727827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.727803 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.728005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.727988 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909037 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909100 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wt2t\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909168 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909211 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909336 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909422 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909663 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.923946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.909798 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.944013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.943978 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.953748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.944415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.965222 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.958291 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.965904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.965879 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.966970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.966941 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:46:09.971965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.971333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:09.987564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.987435 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.987818 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:09.987795 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:10.054398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.054362 4918 generic.go:356] "Generic (PLEG): container finished" podID="70df8927-e3f6-4081-9fa4-8170096e7664" containerID="3202651ea4e6f2c0647c9e18fb7209852b7bff5e7a7fb6ab267b39954b073b6c" exitCode=0 Oct 30 21:46:10.065376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.065340 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.117779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.092225 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.142022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.135684 4918 generic.go:356] "Generic (PLEG): container finished" podID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" containerID="238b044c4c3bd07fcbc76e88216a9f4278a7395f90bf38561b18d345dfa8b3a2" exitCode=137 Oct 30 21:46:10.142022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.138172 4918 generic.go:356] "Generic (PLEG): container finished" podID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" containerID="aff17963b84ecae7b9f4ff87c0421410996e53daee668524e51e3482295f88a1" exitCode=137 Oct 30 21:46:10.146482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.146448 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" event={"ID":"70df8927-e3f6-4081-9fa4-8170096e7664","Type":"ContainerDied","Data":"3202651ea4e6f2c0647c9e18fb7209852b7bff5e7a7fb6ab267b39954b073b6c"} Oct 30 21:46:10.146837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.146820 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:46:10.172595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.172554 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:46:10.177995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.177969 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.183677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.181971 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.210108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.210062 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wt2t\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.298810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.298311 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\") pod \"e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.328082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.325077 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:46:10.356876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.350947 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:46:10.414725 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.411830 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:46:10.499103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.468803 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:10.561143 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.561041 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:46:10.630497 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.630465 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.636285 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636257 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636409 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636338 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636411 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636470 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636518 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636643 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636625 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636696 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636671 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636830 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.636903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.636870 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\") pod \"70df8927-e3f6-4081-9fa4-8170096e7664\" (UID: \"70df8927-e3f6-4081-9fa4-8170096e7664\") " Oct 30 21:46:10.640384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.640351 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.640628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.640599 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.642482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.642443 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.642678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.642652 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.643014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.642989 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.646963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.646082 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.651009 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.650983 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:46:10.675474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.675442 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.680943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.680898 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "kube-api-access-bqmv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.689305 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.689282 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy" pod "70df8927-e3f6-4081-9fa4-8170096e7664" (UID: "70df8927-e3f6-4081-9fa4-8170096e7664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.707990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.707928 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:46:10.743744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.743449 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.743744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.743529 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.743744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.743566 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.743744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.743634 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\" (UniqueName: \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\") pod \"de063a53-ee2e-48a6-a9bb-41a8eebbb126\" (UID: \"de063a53-ee2e-48a6-a9bb-41a8eebbb126\") " Oct 30 21:46:10.744216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744184 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.744343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744232 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.744343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744279 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.744463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744361 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.744518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744503 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.744591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.744574 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\") pod \"608e7593-1fde-41c9-aa03-8a7827b29571\" (UID: \"608e7593-1fde-41c9-aa03-8a7827b29571\") " Oct 30 21:46:10.745425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745402 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmv2\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-kube-api-access-bqmv2\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745428 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745439 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745449 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745458 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70df8927-e3f6-4081-9fa4-8170096e7664-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745466 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70df8927-e3f6-4081-9fa4-8170096e7664-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745475 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745484 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70df8927-e3f6-4081-9fa4-8170096e7664-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.745701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745492 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70df8927-e3f6-4081-9fa4-8170096e7664-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.748984 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.745788 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.749353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.749323 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.751701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.751675 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.752853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.752820 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.753289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.753231 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.756826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.756803 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:46:10.757450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.757423 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.761801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.761754 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.790103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.789993 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "kube-api-access-tzb5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.790492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.790458 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh" pod "de063a53-ee2e-48a6-a9bb-41a8eebbb126" (UID: "de063a53-ee2e-48a6-a9bb-41a8eebbb126"). InnerVolumeSpecName "kube-api-access-wdcxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.792691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.792650 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token" pod "608e7593-1fde-41c9-aa03-8a7827b29571" (UID: "608e7593-1fde-41c9-aa03-8a7827b29571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.805821 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:10.802383 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11385-d18b3b\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:46:10.846081 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846051 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\" (UniqueName: \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\") pod \"1dd3742b-be57-4a7c-9212-c3024f6e2ce2\" (UID: \"1dd3742b-be57-4a7c-9212-c3024f6e2ce2\") " Oct 30 21:46:10.846940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846893 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.846940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846914 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.846940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846923 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e7593-1fde-41c9-aa03-8a7827b29571-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.846940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846932 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.846940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846941 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e7593-1fde-41c9-aa03-8a7827b29571-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.847302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846950 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcxh\" (UniqueName: \"kubernetes.io/projected/de063a53-ee2e-48a6-a9bb-41a8eebbb126-kube-api-access-wdcxh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.847302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846963 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzb5r\" (UniqueName: \"kubernetes.io/projected/608e7593-1fde-41c9-aa03-8a7827b29571-kube-api-access-tzb5r\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.847302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846972 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.847302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846980 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.847302 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.846988 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e7593-1fde-41c9-aa03-8a7827b29571-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.848291 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.848265 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq" pod "1dd3742b-be57-4a7c-9212-c3024f6e2ce2" (UID: "1dd3742b-be57-4a7c-9212-c3024f6e2ce2"). InnerVolumeSpecName "kube-api-access-fqkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.948756 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:10.948724 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkpq\" (UniqueName: \"kubernetes.io/projected/1dd3742b-be57-4a7c-9212-c3024f6e2ce2-kube-api-access-fqkpq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.026909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.026878 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:46:11.028426 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:46:11.028397 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11939dde_6cbd_4750_9059_802b13696e34.slice/crio-2b6ad5ebde0300ba4ccb065f4b71c1faec37b1f884ed68b9e2dbf1bc9855eb69 WatchSource:0}: Error finding container 2b6ad5ebde0300ba4ccb065f4b71c1faec37b1f884ed68b9e2dbf1bc9855eb69: Status 404 returned error can't find the container with id 2b6ad5ebde0300ba4ccb065f4b71c1faec37b1f884ed68b9e2dbf1bc9855eb69 Oct 30 21:46:11.031241 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.031205 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:11.031329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.031270 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:11.031329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.031298 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:11.150408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.150370 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" event={"ID":"70df8927-e3f6-4081-9fa4-8170096e7664","Type":"ContainerDied","Data":"200faac263d13f02dc0ab9aacfe1fdf4317aecc396350804a9fcc3058b7ae087"} Oct 30 21:46:11.150569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.150437 4918 scope.go:122] "RemoveContainer" containerID="3202651ea4e6f2c0647c9e18fb7209852b7bff5e7a7fb6ab267b39954b073b6c" Oct 30 21:46:11.150642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.150623 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6" Oct 30 21:46:11.154962 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.154935 4918 generic.go:356] "Generic (PLEG): container finished" podID="608e7593-1fde-41c9-aa03-8a7827b29571" containerID="fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f" exitCode=0 Oct 30 21:46:11.155155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.155141 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" Oct 30 21:46:11.155299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.155283 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" event={"ID":"608e7593-1fde-41c9-aa03-8a7827b29571","Type":"ContainerDied","Data":"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f"} Oct 30 21:46:11.155372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.155321 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq" event={"ID":"608e7593-1fde-41c9-aa03-8a7827b29571","Type":"ContainerDied","Data":"1bd6382a08b52eb10c996f87de395931b26cd33a475cadeaf6360054c0f30847"} Oct 30 21:46:11.158394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.157954 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:46:11.162099 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.161291 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:46:11.167833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.167766 4918 scope.go:122] "RemoveContainer" containerID="fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f" Oct 30 21:46:11.172790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.171252 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerStarted","Data":"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50"} Oct 30 21:46:11.172790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.171299 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerStarted","Data":"2b6ad5ebde0300ba4ccb065f4b71c1faec37b1f884ed68b9e2dbf1bc9855eb69"} Oct 30 21:46:11.188907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.188888 4918 scope.go:122] "RemoveContainer" containerID="fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f" Oct 30 21:46:11.189144 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:11.189125 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f\": container with ID starting with fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f not found: ID does not exist" containerID="fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f" Oct 30 21:46:11.189232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.189151 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f"} err="failed to get container status \"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f\": rpc error: code = NotFound desc = could not find container \"fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f\": container with ID starting with fe5d01823d05749c3f7931df8cf9de8a90066924c653c83ddaa33681294e014f not found: ID does not exist" Oct 30 21:46:11.189232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.189181 4918 scope.go:122] "RemoveContainer" containerID="238b044c4c3bd07fcbc76e88216a9f4278a7395f90bf38561b18d345dfa8b3a2" Oct 30 21:46:11.208934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.208153 4918 scope.go:122] "RemoveContainer" containerID="aff17963b84ecae7b9f4ff87c0421410996e53daee668524e51e3482295f88a1" Oct 30 21:46:11.432171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.432120 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podStartSLOduration=4.432103261 podStartE2EDuration="4.432103261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.429854282 +0000 UTC m=+1952.123336519" watchObservedRunningTime="2026-10-30 21:46:11.432103261 +0000 UTC m=+1952.125585501" Oct 30 21:46:11.433209 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.433186 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" path="/var/lib/kubelet/pods/1dd3742b-be57-4a7c-9212-c3024f6e2ce2/volumes" Oct 30 21:46:11.433635 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.433622 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" path="/var/lib/kubelet/pods/de063a53-ee2e-48a6-a9bb-41a8eebbb126/volumes" Oct 30 21:46:11.469334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.469302 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:11.553938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.553901 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:46:11.559030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.559005 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4fgdq"] Oct 30 21:46:11.710816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.710733 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:46:11.756313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:11.756281 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lq6p6"] Oct 30 21:46:11.760059 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:11.760024 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11382-3c5c1a\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="1dd3742b-be57-4a7c-9212-c3024f6e2ce2" pod="opendatahub/test-curl-11382-3c5c1a" Oct 30 21:46:11.808216 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:11.808176 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11385-d18b3b\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="de063a53-ee2e-48a6-a9bb-41a8eebbb126" pod="opendatahub/test-curl-11385-d18b3b" Oct 30 21:46:12.474956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:12.474926 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:12.502477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:12.502437 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.690300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:12.690267 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.195916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.195883 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:13.197068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.197045 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:46:13.435758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.435725 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" path="/var/lib/kubelet/pods/608e7593-1fde-41c9-aa03-8a7827b29571/volumes" Oct 30 21:46:13.436330 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.436314 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" path="/var/lib/kubelet/pods/70df8927-e3f6-4081-9fa4-8170096e7664/volumes" Oct 30 21:46:13.509622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.509517 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.707411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:13.707376 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.172434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:15.172390 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.421492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:15.421457 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.620042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:15.620011 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.805673 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:15.805641 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:46:15.808569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:15.808525 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.014149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.014069 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:46:16.014910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.014887 4918 cpu_manager.go:414] "containerMap: removing container" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" Oct 30 21:46:16.014910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.014910 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" Oct 30 21:46:16.015098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.014965 4918 cpu_manager.go:414] "containerMap: removing container" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" Oct 30 21:46:16.015098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.014973 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" Oct 30 21:46:16.015284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.015261 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="70df8927-e3f6-4081-9fa4-8170096e7664" containerName="istio-proxy" Oct 30 21:46:16.015396 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.015297 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e7593-1fde-41c9-aa03-8a7827b29571" containerName="istio-proxy" Oct 30 21:46:16.020132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.020105 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.020811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.020792 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.091517 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.091482 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:46:16.118642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.118617 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-899a6bb1-openshift-default-dockercfg-xg72j\"" Oct 30 21:46:16.138499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.138469 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.138643 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.138521 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.138818 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.138793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vx86\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.138909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.138889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.138995 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.138978 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.139092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.139072 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.139249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.139231 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.139306 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.139295 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.139401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.139384 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240368 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240444 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240489 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240511 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240613 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vx86\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240661 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240750 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.240827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240819 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.240891 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.241026 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.241085 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.241235 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241424 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.241298 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.241641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.241617 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.242990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.242963 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.243219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.243198 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.299376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.299313 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.299524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.299512 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vx86\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\") pod \"e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.335451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.335428 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:16.342904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.342872 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.605344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.605239 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.872169 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:46:16.872136 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda405d997_14fa_4a72_a119_a39897fee106.slice/crio-21395f32ce61a21c33066a1703a8921d54cfdd4d536763f23e5c896c452dbbdc WatchSource:0}: Error finding container 21395f32ce61a21c33066a1703a8921d54cfdd4d536763f23e5c896c452dbbdc: Status 404 returned error can't find the container with id 21395f32ce61a21c33066a1703a8921d54cfdd4d536763f23e5c896c452dbbdc Oct 30 21:46:16.873656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.873627 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:46:16.874603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.874573 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:16.874695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.874635 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:16.874695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:16.874663 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:17.236986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:17.236869 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerStarted","Data":"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d"} Oct 30 21:46:17.236986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:17.236964 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerStarted","Data":"21395f32ce61a21c33066a1703a8921d54cfdd4d536763f23e5c896c452dbbdc"} Oct 30 21:46:17.336455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:17.336429 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:17.341946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:17.341924 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:17.485981 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:17.485931 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podStartSLOduration=2.485914928 podStartE2EDuration="2.485914928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.364987128 +0000 UTC m=+1958.058469363" watchObservedRunningTime="2026-10-30 21:46:17.485914928 +0000 UTC m=+1958.179397164" Oct 30 21:46:18.246248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:18.246222 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:18.247057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:18.247037 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:46:19.497459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:19.497422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:19.706068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:19.706033 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:21.124382 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:21.124344 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:22.144919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:22.144879 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:22.346068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:22.346037 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.415771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:23.414246 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.737499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:23.737379 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:24.725020 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:24.724965 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:25.082927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:25.082821 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.028618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:26.028587 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.279737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:26.279620 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:30.621402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:30.621366 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:32.949938 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:32.949905 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:46:32.954533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:32.954501 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.023321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.023294 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:46:33.095674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.095640 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.095674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.095682 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.095953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.095859 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.095953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.095917 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.096071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.096042 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.096166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.096153 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.096204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.096189 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.096262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.096250 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.096295 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.096285 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c767h\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197288 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197351 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197382 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197448 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c767h\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197579 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197615 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197729 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197811 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197818 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197855 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.197966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.197904 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.198170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.198105 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.198619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.198574 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.200283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.200218 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.200283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.200260 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.257325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.257301 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.257614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.257594 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c767h\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.270562 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.270539 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:33.706297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.706263 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:46:33.706760 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:46:33.706731 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac21b833_45e1_43b7_a112_0e9e6cad9332.slice/crio-485cd6f61cbbd07fadd5536a8ab6955ed5b6ea1ded8e6b44a9045559f4deab32 WatchSource:0}: Error finding container 485cd6f61cbbd07fadd5536a8ab6955ed5b6ea1ded8e6b44a9045559f4deab32: Status 404 returned error can't find the container with id 485cd6f61cbbd07fadd5536a8ab6955ed5b6ea1ded8e6b44a9045559f4deab32 Oct 30 21:46:33.709427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.709386 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:33.709505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.709474 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:33.709555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:33.709540 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:34.398954 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:34.398920 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" event={"ID":"ac21b833-45e1-43b7-a112-0e9e6cad9332","Type":"ContainerStarted","Data":"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76"} Oct 30 21:46:34.399450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:34.398980 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" event={"ID":"ac21b833-45e1-43b7-a112-0e9e6cad9332","Type":"ContainerStarted","Data":"485cd6f61cbbd07fadd5536a8ab6955ed5b6ea1ded8e6b44a9045559f4deab32"} Oct 30 21:46:34.619265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:34.619212 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podStartSLOduration=2.619196925 podStartE2EDuration="2.619196925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:34.569746676 +0000 UTC m=+1975.263228918" watchObservedRunningTime="2026-10-30 21:46:34.619196925 +0000 UTC m=+1975.312679160" Oct 30 21:46:35.283728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.273001 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:35.293729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.284824 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:35.293729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.284885 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:35.519654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.519607 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:35.718638 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.718603 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:35.925156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:35.925097 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:36.274731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:36.274063 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:36.274731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:36.274138 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:36.523731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:36.514983 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:37.096473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:37.088131 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:37.289730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:37.288064 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:37.289730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:37.288154 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:38.282730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:38.274224 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:38.283269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:38.282781 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:39.282733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:39.279089 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:39.282733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:39.279167 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:39.764792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:39.764744 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:39.892732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:39.891723 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:39.896728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:39.892963 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.006680 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.006631 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:46:40.049078 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.048965 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 30 21:46:40.104003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.103961 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.283859 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.283817 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:40.284945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.284910 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:40.393732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.387798 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.713787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.709845 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:40.784732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.784007 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:46:40.788727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.784957 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:46:40.907732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.905845 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:40.907732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.906663 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:46:40.935738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.908494 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.935738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.908942 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:46:40.935738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.911830 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:40.935738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.911893 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.935738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:40.916988 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:41.257732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.237929 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 21:46:41.260070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.260042 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:46:41.288999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.288963 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:41.361207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.361106 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:41.501930 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.451666 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.809993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.809944 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.936736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:41.928384 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.058176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.056856 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.058176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.057029 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:42.058176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.057080 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.126799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.125543 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:46:42.126799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.125591 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 21:46:42.173544 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.165438 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.232142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.232093 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:46:42.233317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.233293 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:46:42.233568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.233262 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vmmk5\"" Oct 30 21:46:42.233717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.233691 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:46:42.292240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.292098 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.297870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.297842 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:42.348736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.348631 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:42.368730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.367093 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.388795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.367276 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.389121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.389092 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.389354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.389320 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.389591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.389571 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7ck\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.405439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.405406 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.405777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.405743 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.487881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.487829 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:42.488185 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.488139 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.508987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.467874 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.709977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.709840 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:42.709977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.709927 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.837648 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.818364 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927472 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927554 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927599 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927680 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7ck\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927794 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927852 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.955262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.927987 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:42.982527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:42.973293 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.141739 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.133406 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.167845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.167642 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.343824 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.300971 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.469445 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.469412 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:43.469736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.469684 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:43.504967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.482736 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.540726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.515663 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.540726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.534825 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 21:46:43.540726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.538050 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:43.540726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.538159 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.690122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.653225 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.690122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.653459 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.745936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.717140 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7ck\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.745936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.717278 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:43.745936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.717324 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.747132 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.747101 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z6ltt\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.904736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.904577 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 21:46:43.910870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.910836 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.919122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.915519 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:43.975499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:43.954976 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:44.021372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.021251 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 30 21:46:44.177545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.177215 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" podUID="8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.215740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.214180 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt" Oct 30 21:46:44.225489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.225465 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 21:46:44.273952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.273924 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:46:44.276144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.276113 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 21:46:44.306470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.306425 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.382098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.381633 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 21:46:44.387829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.387799 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 21:46:44.408695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.408674 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:46:44.788256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.784450 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 21:46:44.932859 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:44.932825 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:45.017866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:45.017821 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" event={"ID":"417f64fe-3c08-4fd3-9d06-52b3a278ae94","Type":"ContainerStarted","Data":"c74ec5b6ca4efec4897c6c738e2c4d793a109ddf97800269356fc19efeac8340"} Oct 30 21:46:45.181681 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:45.181643 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:45.882766 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:46:45.880698 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:46:45.962177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:45.961889 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:45.962480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:45.962459 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:45.966061 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:45.966035 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:46:49.035447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:49.035411 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:49.156333 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:49.156296 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:49.156440 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:49.156364 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:50.110117 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:50.110072 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" event={"ID":"417f64fe-3c08-4fd3-9d06-52b3a278ae94","Type":"ContainerStarted","Data":"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087"} Oct 30 21:46:50.113146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:50.110152 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:50.113146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:50.112746 4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z6ltt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:50.113146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:50.112798 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:50.307030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:50.306967 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" podStartSLOduration=4.945297587 podStartE2EDuration="9.306953103s" totalImagesPullingTime="4.361655516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:50.276002625 +0000 UTC m=+1990.969484861" watchObservedRunningTime="2026-10-30 21:46:50.306953103 +0000 UTC m=+1991.000435338" Oct 30 21:46:51.120063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:51.120034 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 21:46:53.621082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:53.619059 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:53.828724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:53.826322 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:54.013485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:54.013405 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:58.646586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:46:58.646549 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:00.452589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:00.452550 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.105779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:01.105734 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.314380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:01.314350 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:02.054149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:02.054112 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:02.257239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:02.257201 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.286909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:03.286878 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.520339 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:03.520305 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.807079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:05.807046 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:06.001478 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:06.001391 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.251666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:08.251609 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:11.412551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:11.412514 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:11.614994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:11.614957 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:12.176788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:12.135062 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:12.392072 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:12.392030 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.081814 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:13.081730 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.280217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:13.280177 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.486895 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:13.486856 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.889821 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:47:15.889089 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:47:18.384798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:18.384756 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.610761 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:18.610693 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:23.009421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:23.009330 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:25.129693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:25.129657 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.638296 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:27.638254 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.248778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:32.248737 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.693288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:36.693250 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.902195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:36.902156 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.169304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:41.169271 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.410590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:41.410553 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.600929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:41.600842 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.803401 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:47:45.803364 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:47:46.033842 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:46.033780 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:50.610758 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:50.610288 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.420025 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:55.419969 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.647642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:57.647603 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.842098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:47:57.842067 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:00.220290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:00.220254 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:05.566487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.566447 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:48:05.573270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.573241 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.625344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.625307 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-364e1518-serving-cert\"" Oct 30 21:48:05.637594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.637553 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.637817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.637634 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.637817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.637736 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.638010 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.637991 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpz2\" (UniqueName: \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.738993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.738925 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.739204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.739049 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.739406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.739381 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.739536 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:48:05.739513 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-serving-cert: secret "maas-api-e2e-mt-364e1518-serving-cert" not found Oct 30 21:48:05.739595 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:48:05.739583 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls podName:86a7fb59-86f2-4ea7-adff-10a61b65d055 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:06.23956581 +0000 UTC m=+2066.933048028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls") pod "maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" (UID: "86a7fb59-86f2-4ea7-adff-10a61b65d055") : secret "maas-api-e2e-mt-364e1518-serving-cert" not found Oct 30 21:48:05.739660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.739610 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.739960 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.739941 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpz2\" (UniqueName: \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.741797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.741746 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.834667 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.834600 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpz2\" (UniqueName: \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:05.838478 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.838345 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:48:05.929870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:05.912054 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:06.115835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:06.113598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:06.336433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:06.336375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:06.339794 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:06.339764 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") pod \"maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:06.542393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:06.542302 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:07.026396 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:48:07.026362 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a7fb59_86f2_4ea7_adff_10a61b65d055.slice/crio-0179ce9ff68920647125dc6284d728ac657a9172e09b879bea84cb93c6c04ec3 WatchSource:0}: Error finding container 0179ce9ff68920647125dc6284d728ac657a9172e09b879bea84cb93c6c04ec3: Status 404 returned error can't find the container with id 0179ce9ff68920647125dc6284d728ac657a9172e09b879bea84cb93c6c04ec3 Oct 30 21:48:07.029871 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:07.029844 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:48:07.233015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:07.232975 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" event={"ID":"86a7fb59-86f2-4ea7-adff-10a61b65d055","Type":"ContainerStarted","Data":"0179ce9ff68920647125dc6284d728ac657a9172e09b879bea84cb93c6c04ec3"} Oct 30 21:48:08.242213 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:08.242110 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" event={"ID":"86a7fb59-86f2-4ea7-adff-10a61b65d055","Type":"ContainerStarted","Data":"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea"} Oct 30 21:48:08.242670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:08.242223 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:08.362123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:08.362058 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" podStartSLOduration=2.464734914 podStartE2EDuration="3.362038147s" totalImagesPullingTime="897.303233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:08.356560286 +0000 UTC m=+2069.050042520" watchObservedRunningTime="2026-10-30 21:48:08.362038147 +0000 UTC m=+2069.055520385" Oct 30 21:48:09.757557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:09.757524 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.289600 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:11.289561 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.294259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.294175 4918 generic.go:356] "Generic (PLEG): container finished" podID="a405d997-14fa-4a72-a119-a39897fee106" containerID="0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d" exitCode=0 Oct 30 21:48:12.294259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.294244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerDied","Data":"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d"} Oct 30 21:48:12.294845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.294747 4918 scope.go:122] "RemoveContainer" containerID="0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d" Oct 30 21:48:12.296830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.296794 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.296953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.296889 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.296953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.296934 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.560119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.560073 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.774555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:12.774458 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:13.200271 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.200219 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:13.200545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.200290 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:13.315188 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.315147 4918 generic.go:356] "Generic (PLEG): container finished" podID="11939dde-6cbd-4750-9059-802b13696e34" containerID="97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50" exitCode=0 Oct 30 21:48:13.315653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.315206 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerDied","Data":"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50"} Oct 30 21:48:13.315825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.315807 4918 scope.go:122] "RemoveContainer" containerID="97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50" Oct 30 21:48:13.317803 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.317767 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:13.317910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.317845 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:13.317910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.317878 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:13.319498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.319474 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerStarted","Data":"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f"} Oct 30 21:48:13.335925 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.335903 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:48:13.336094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.336076 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:13.336141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:13.336129 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:14.301727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.300871 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:48:14.334569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.334539 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerStarted","Data":"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261"} Oct 30 21:48:14.335992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.335968 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:14.336142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.336008 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:14.469239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.469210 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:48:14.469421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.469313 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:14.469421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:14.469351 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:15.335908 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:15.335874 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:15.336433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:15.336067 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:15.469137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:15.469107 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:15.469304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:15.469161 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:15.858369 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:48:15.858329 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:48:16.307731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.307197 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.336415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.336381 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:16.336916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.336438 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:16.485854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.485806 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:16.487453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.487415 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:16.630475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:16.630391 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.337077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:17.337037 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:17.337508 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:17.337100 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:17.473833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:17.473798 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:17.474099 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:17.474072 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:18.246355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:18.246323 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:48:18.336200 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:18.336167 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:48:18.336442 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:18.336422 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:48:18.469868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:18.469836 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:18.470435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:18.470407 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:19.341210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:19.341175 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:48:19.342343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:19.342311 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:48:19.469381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:19.469353 4918 patch_prober.go:29] interesting pod/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:19.469546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:19.469398 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 21:48:20.474772 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:20.474740 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:48:20.475283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:20.475270 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:48:20.476110 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:20.476088 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:48:22.149686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:22.149640 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.134522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:24.134484 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.340011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:24.339981 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:27.247983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:27.247939 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:32.116358 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:32.116314 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:34.357415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:34.357379 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:34.541323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:34.541287 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:37.071967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:37.071927 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:39.734242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.734210 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:48:39.737219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.737189 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:48:39.748780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.748758 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:48:39.750780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.750761 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:48:39.754885 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.754864 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:48:39.757150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:39.757128 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:48:41.498316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:41.498281 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:41.741668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:41.741628 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:45.905297 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:48:45.905258 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:48:46.548364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:46.548326 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:48.790177 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:48.790143 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:48.997150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:48.997119 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:49.437069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:49.437031 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.538816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:51.538777 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:55.879092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:55.879048 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:56.077910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:56.077804 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.350409 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:57.350334 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.613853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:57.613821 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.799035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:57.798994 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.131064 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:58.131010 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.873490 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:58.873456 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.074847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:59.074809 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.286868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:59.286769 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.782269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:59.782232 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.009730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:48:59.996377 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:01.548937 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:01.548896 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:01.761970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:01.761934 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:02.386444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:02.386408 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.566399 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:03.566345 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.758807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:03.758772 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.955856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:03.955818 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.271575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:04.269089 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.326113 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:05.326061 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.616464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:05.616432 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:06.249481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:06.249448 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:06.491211 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:06.491174 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:07.404137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:07.404099 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:49:07.404393 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:07.404361 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" containerID="cri-o://72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261" gracePeriod=30 Oct 30 21:49:07.415355 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:07.415326 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.403807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:08.403676 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.675443 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:08.675348 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.416195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:09.416153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:12.733639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.733612 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:49:12.824616 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824588 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.824846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824643 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.824846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824778 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.824846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824832 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824890 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.824982 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825062 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825057 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:12.825141 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825096 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:12.825281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825175 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825214 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\") pod \"11939dde-6cbd-4750-9059-802b13696e34\" (UID: \"11939dde-6cbd-4750-9059-802b13696e34\") " Oct 30 21:49:12.825281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825268 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:12.825522 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825502 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:12.825768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825743 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:12.825993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.825976 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.826065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.826001 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11939dde-6cbd-4750-9059-802b13696e34-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.826065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.826016 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.826065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.826030 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.826065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.826044 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.827176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.827148 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:12.827293 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.827242 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:12.827360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.827330 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:12.827360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.827337 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t" pod "11939dde-6cbd-4750-9059-802b13696e34" (UID: "11939dde-6cbd-4750-9059-802b13696e34"). InnerVolumeSpecName "kube-api-access-2wt2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:12.927298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.927264 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wt2t\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-kube-api-access-2wt2t\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.927298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.927295 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11939dde-6cbd-4750-9059-802b13696e34-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.927298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.927304 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11939dde-6cbd-4750-9059-802b13696e34-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.927598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:12.927313 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11939dde-6cbd-4750-9059-802b13696e34-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:13.106914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.106833 4918 generic.go:356] "Generic (PLEG): container finished" podID="11939dde-6cbd-4750-9059-802b13696e34" containerID="72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261" exitCode=0 Oct 30 21:49:13.106914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.106876 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" Oct 30 21:49:13.107083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.106919 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerDied","Data":"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261"} Oct 30 21:49:13.107083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.106975 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks" event={"ID":"11939dde-6cbd-4750-9059-802b13696e34","Type":"ContainerDied","Data":"2b6ad5ebde0300ba4ccb065f4b71c1faec37b1f884ed68b9e2dbf1bc9855eb69"} Oct 30 21:49:13.107083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.107001 4918 scope.go:122] "RemoveContainer" containerID="72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261" Oct 30 21:49:13.119671 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.119648 4918 scope.go:122] "RemoveContainer" containerID="97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50" Oct 30 21:49:13.129623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.129602 4918 scope.go:122] "RemoveContainer" containerID="72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261" Oct 30 21:49:13.129927 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:13.129908 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261\": container with ID starting with 72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261 not found: ID does not exist" containerID="72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261" Oct 30 21:49:13.129998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.129936 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261"} err="failed to get container status \"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261\": rpc error: code = NotFound desc = could not find container \"72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261\": container with ID starting with 72fd8d28645542bd69c8560f45e44141881e75c411c4c445aa0419c77be84261 not found: ID does not exist" Oct 30 21:49:13.129998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.129952 4918 scope.go:122] "RemoveContainer" containerID="97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50" Oct 30 21:49:13.130183 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:13.130167 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50\": container with ID starting with 97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50 not found: ID does not exist" containerID="97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50" Oct 30 21:49:13.130223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.130188 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50"} err="failed to get container status \"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50\": rpc error: code = NotFound desc = could not find container \"97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50\": container with ID starting with 97d439c0c7f44555258536a24691971809b4f8757f27124ca327025cdd76ef50 not found: ID does not exist" Oct 30 21:49:13.267082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.267049 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:49:13.271857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.271834 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a84c1592-openshift-default-6cfbbfdc4c-4htks"] Oct 30 21:49:13.430298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.430267 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11939dde-6cbd-4750-9059-802b13696e34" path="/var/lib/kubelet/pods/11939dde-6cbd-4750-9059-802b13696e34/volumes" Oct 30 21:49:13.742268 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.742185 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:13.937152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:13.937116 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:14.888193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:14.888156 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.075976 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.075940 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.266079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.266000 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.507135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.507101 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.698563 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.698522 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.804570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.804527 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:49:15.819766 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.807094 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" containerID="cri-o://5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f" gracePeriod=30 Oct 30 21:49:15.838109 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:15.838064 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:49:15.903733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:15.897669 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:16.862129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:16.862095 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:17.059258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:17.059220 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:17.813362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:17.813322 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:19.343691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:19.343653 4918 patch_prober.go:29] interesting pod/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:19.344126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:19.343747 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:20.556372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:20.556325 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:21.157946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.157845 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:49:21.179943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.179911 4918 generic.go:356] "Generic (PLEG): container finished" podID="a405d997-14fa-4a72-a119-a39897fee106" containerID="5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f" exitCode=0 Oct 30 21:49:21.180103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.180018 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerDied","Data":"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f"} Oct 30 21:49:21.180103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.180081 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" event={"ID":"a405d997-14fa-4a72-a119-a39897fee106","Type":"ContainerDied","Data":"21395f32ce61a21c33066a1703a8921d54cfdd4d536763f23e5c896c452dbbdc"} Oct 30 21:49:21.180103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.180019 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n" Oct 30 21:49:21.180103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.180100 4918 scope.go:122] "RemoveContainer" containerID="5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f" Oct 30 21:49:21.192951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.192930 4918 scope.go:122] "RemoveContainer" containerID="0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d" Oct 30 21:49:21.203763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.203735 4918 scope.go:122] "RemoveContainer" containerID="5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f" Oct 30 21:49:21.204107 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:21.204081 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f\": container with ID starting with 5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f not found: ID does not exist" containerID="5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f" Oct 30 21:49:21.204185 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.204120 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f"} err="failed to get container status \"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f\": rpc error: code = NotFound desc = could not find container \"5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f\": container with ID starting with 5ed5609424ac3238d8818ee1ef0ae9b9fbde2912aa9dd05d8320075848661e2f not found: ID does not exist" Oct 30 21:49:21.204185 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.204144 4918 scope.go:122] "RemoveContainer" containerID="0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d" Oct 30 21:49:21.204463 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:21.204441 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d\": container with ID starting with 0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d not found: ID does not exist" containerID="0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d" Oct 30 21:49:21.204528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.204473 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d"} err="failed to get container status \"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d\": rpc error: code = NotFound desc = could not find container \"0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d\": container with ID starting with 0a8346a7d0fcc7281637fdc8691847e60cd9e63d17f8e45960923b31434c693d not found: ID does not exist" Oct 30 21:49:21.283111 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283080 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283155 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283209 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283421 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283514 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283557 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:21.283591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283571 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283607 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283605 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:21.283699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283636 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283666 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\") pod \"a405d997-14fa-4a72-a119-a39897fee106\" (UID: \"a405d997-14fa-4a72-a119-a39897fee106\") " Oct 30 21:49:21.283958 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.283762 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:21.284049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284026 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:21.284114 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284068 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:21.284451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284428 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.284553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284453 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.284553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284469 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.284553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284482 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.284553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.284496 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a405d997-14fa-4a72-a119-a39897fee106-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.286312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.286280 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:21.286433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.286394 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:21.286677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.286645 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "kube-api-access-8vx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:21.286806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.286784 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy" pod "a405d997-14fa-4a72-a119-a39897fee106" (UID: "a405d997-14fa-4a72-a119-a39897fee106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:21.386665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.386574 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vx86\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-kube-api-access-8vx86\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.386665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.386616 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a405d997-14fa-4a72-a119-a39897fee106-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.386665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.386629 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a405d997-14fa-4a72-a119-a39897fee106-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.386665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.386643 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a405d997-14fa-4a72-a119-a39897fee106-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:21.593119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.593082 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:49:21.638447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:21.638376 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-899a6bb1-openshift-default-67bd7d5777-mp56n"] Oct 30 21:49:22.001593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:22.001504 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.430915 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:23.430879 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a405d997-14fa-4a72-a119-a39897fee106" path="/var/lib/kubelet/pods/a405d997-14fa-4a72-a119-a39897fee106/volumes" Oct 30 21:49:25.237234 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:25.237193 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:25.432341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:25.432305 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.815737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:29.814910 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.797325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:33.797272 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.308788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:35.308743 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.528443 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:35.528409 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.731990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:35.731956 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:36.414650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:36.414615 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.417892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:37.417852 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.613095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:37.613059 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.958115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:38.958083 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.258890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:39.258813 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.584486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:40.584445 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.875182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:41.875141 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.284639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:44.284605 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.077445 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:45.077406 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.277021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:45.276984 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.766592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:45.766556 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.847491 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:49:45.847456 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:49:46.536894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:46.536861 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.036685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:50.036632 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.432647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:51.432590 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.644989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:51.644953 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.269569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:52.269521 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.711444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:53.711394 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.903153 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:53.903112 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:55.051894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:55.051844 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.702514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:58.702475 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.715856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:49:59.715806 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.371876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.371834 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.376850 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6"] Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378032 4918 cpu_manager.go:414] "containerMap: removing container" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378058 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378109 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378118 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378132 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.378140 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.379377 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.379407 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.382751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.379418 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a405d997-14fa-4a72-a119-a39897fee106" containerName="istio-proxy" Oct 30 21:50:04.387176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.387103 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458244 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458376 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458488 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458591 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33be08a-8641-49fd-9b8e-87231d6d10e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458802 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458781 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdz4\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-kube-api-access-6jdz4\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.458906 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.459019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.458999 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.459073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.459057 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.459159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.459138 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.505863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.500322 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6"] Oct 30 21:50:04.560412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560382 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdz4\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-kube-api-access-6jdz4\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560441 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560476 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560505 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560528 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560660 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560758 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560835 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560820 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560876 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33be08a-8641-49fd-9b8e-87231d6d10e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560921 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.560996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.560979 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.561158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.561028 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.561158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.561070 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.561579 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.561553 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33be08a-8641-49fd-9b8e-87231d6d10e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.563051 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.563031 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.563130 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.563045 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.618663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.618636 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.618845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.618823 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdz4\" (UniqueName: \"kubernetes.io/projected/e33be08a-8641-49fd-9b8e-87231d6d10e1-kube-api-access-6jdz4\") pod \"maas-default-gateway-openshift-default-67bf986f99-hpcg6\" (UID: \"e33be08a-8641-49fd-9b8e-87231d6d10e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:04.710609 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:04.710539 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:05.141118 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:50:05.141091 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33be08a_8641_49fd_9b8e_87231d6d10e1.slice/crio-2a8668b9c110ac199652b1c8fb4a60ea9105aec4bb247ecca303c5bdb76b22d5 WatchSource:0}: Error finding container 2a8668b9c110ac199652b1c8fb4a60ea9105aec4bb247ecca303c5bdb76b22d5: Status 404 returned error can't find the container with id 2a8668b9c110ac199652b1c8fb4a60ea9105aec4bb247ecca303c5bdb76b22d5 Oct 30 21:50:05.141323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.141298 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6"] Oct 30 21:50:05.144125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.144095 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:05.144217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.144158 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:05.144217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.144202 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:05.651240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.651198 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" event={"ID":"e33be08a-8641-49fd-9b8e-87231d6d10e1","Type":"ContainerStarted","Data":"06d15f4dc4cd3b4f2d57d2596ac46df01b3fa298b982281932d0c27ad8ca046a"} Oct 30 21:50:05.651655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.651262 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" event={"ID":"e33be08a-8641-49fd-9b8e-87231d6d10e1","Type":"ContainerStarted","Data":"2a8668b9c110ac199652b1c8fb4a60ea9105aec4bb247ecca303c5bdb76b22d5"} Oct 30 21:50:05.710907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:05.710869 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:06.713650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:06.712378 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:06.713650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:06.712461 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:07.711519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:07.711486 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:07.711752 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:07.711552 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:08.711982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:08.711939 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:08.712487 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:08.712019 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:09.089785 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:09.084125 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.711311 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:09.711272 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:09.711489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:09.711348 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:10.711404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:10.711369 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:10.712041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:10.712004 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:11.711628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:11.711582 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:11.712073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:11.711666 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:12.711349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:12.711310 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:50:12.711558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:12.711390 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:50:13.845124 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:13.845092 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:13.846308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:13.846276 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:13.847057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:13.847036 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" Oct 30 21:50:13.999273 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:13.999213 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podStartSLOduration=9.999191939 podStartE2EDuration="9.999191939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:05.864020121 +0000 UTC m=+2186.557502358" watchObservedRunningTime="2026-10-30 21:50:13.999191939 +0000 UTC m=+2194.692674176" Oct 30 21:50:14.055810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:14.055775 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.822360 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:50:15.822322 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:50:18.740029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:18.739990 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:20.880148 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:20.880068 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:50:20.880673 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:20.880313 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" containerID="cri-o://923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc" gracePeriod=30 Oct 30 21:50:20.926693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:20.926653 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:50:20.926957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:20.926924 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" containerID="cri-o://f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794" gracePeriod=30 Oct 30 21:50:22.461969 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:22.461931 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:22.462386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:22.461994 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:23.432548 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:23.432506 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.178820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.178791 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:50:26.248301 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.248277 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:50:26.281975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.281950 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282114 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.281993 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282114 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282040 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282128 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282148 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282176 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282208 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282264 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282299 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282327 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282398 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282439 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282317 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282436 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282447 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282479 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282547 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282555 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282601 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282641 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282692 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\") pod \"2db2f944-3425-4d63-be41-4252daa4b028\" (UID: \"2db2f944-3425-4d63-be41-4252daa4b028\") " Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282785 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:26.282798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.282801 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\") pod \"241d1ba1-dcc2-46df-a016-6d021f131514\" (UID: \"241d1ba1-dcc2-46df-a016-6d021f131514\") " Oct 30 21:50:26.283385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.283259 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.283385 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.283264 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.283488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.283449 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.285077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285039 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285082 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db2f944-3425-4d63-be41-4252daa4b028-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285096 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285108 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241d1ba1-dcc2-46df-a016-6d021f131514-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285122 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285137 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285151 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285166 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285180 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.285662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285247 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.285662 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285385 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.285793 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285693 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.285888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.285867 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.286183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.286161 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.286292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.286161 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:26.286364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.286303 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "kube-api-access-g9dts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.287218 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.287190 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo" pod "241d1ba1-dcc2-46df-a016-6d021f131514" (UID: "241d1ba1-dcc2-46df-a016-6d021f131514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:26.287505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.287488 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7" pod "2db2f944-3425-4d63-be41-4252daa4b028" (UID: "2db2f944-3425-4d63-be41-4252daa4b028"). InnerVolumeSpecName "kube-api-access-dvxt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.386914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386884 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.386914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386914 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dts\" (UniqueName: \"kubernetes.io/projected/241d1ba1-dcc2-46df-a016-6d021f131514-kube-api-access-g9dts\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386929 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db2f944-3425-4d63-be41-4252daa4b028-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386944 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386962 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241d1ba1-dcc2-46df-a016-6d021f131514-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386973 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241d1ba1-dcc2-46df-a016-6d021f131514-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386984 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxt7\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-kube-api-access-dvxt7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.386995 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db2f944-3425-4d63-be41-4252daa4b028-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.387170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.387006 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db2f944-3425-4d63-be41-4252daa4b028-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.886392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.886360 4918 generic.go:356] "Generic (PLEG): container finished" podID="241d1ba1-dcc2-46df-a016-6d021f131514" containerID="f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794" exitCode=0 Oct 30 21:50:26.886590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.886419 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" Oct 30 21:50:26.886590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.886451 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" event={"ID":"241d1ba1-dcc2-46df-a016-6d021f131514","Type":"ContainerDied","Data":"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794"} Oct 30 21:50:26.886590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.886540 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl" event={"ID":"241d1ba1-dcc2-46df-a016-6d021f131514","Type":"ContainerDied","Data":"12160fd7ebc5de5aacf0ef917658fca0eb75b2f1e746cff7fe8b28ee20609f53"} Oct 30 21:50:26.886590 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.886569 4918 scope.go:122] "RemoveContainer" containerID="f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794" Oct 30 21:50:26.888118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.888079 4918 generic.go:356] "Generic (PLEG): container finished" podID="2db2f944-3425-4d63-be41-4252daa4b028" containerID="923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc" exitCode=0 Oct 30 21:50:26.888226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.888152 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" Oct 30 21:50:26.888292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.888273 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" event={"ID":"2db2f944-3425-4d63-be41-4252daa4b028","Type":"ContainerDied","Data":"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc"} Oct 30 21:50:26.888347 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.888310 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22" event={"ID":"2db2f944-3425-4d63-be41-4252daa4b028","Type":"ContainerDied","Data":"e3d71158d6144b1818ab25339db065fd99a8bd3d75152ded13d95d359c9857f3"} Oct 30 21:50:26.898261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.898243 4918 scope.go:122] "RemoveContainer" containerID="f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794" Oct 30 21:50:26.898523 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:50:26.898502 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794\": container with ID starting with f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794 not found: ID does not exist" containerID="f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794" Oct 30 21:50:26.898608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.898533 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794"} err="failed to get container status \"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794\": rpc error: code = NotFound desc = could not find container \"f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794\": container with ID starting with f64a4ff1890d0c082b092ad4d51db9272ee7615a5ab417a79f6cd85a233ff794 not found: ID does not exist" Oct 30 21:50:26.898608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.898556 4918 scope.go:122] "RemoveContainer" containerID="923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc" Oct 30 21:50:26.920595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.920578 4918 scope.go:122] "RemoveContainer" containerID="923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc" Oct 30 21:50:26.920881 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:50:26.920857 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc\": container with ID starting with 923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc not found: ID does not exist" containerID="923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc" Oct 30 21:50:26.920952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:26.920885 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc"} err="failed to get container status \"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc\": rpc error: code = NotFound desc = could not find container \"923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc\": container with ID starting with 923f0de828b10670963cdff021ffe315df6f2355d6802bf99ef56b4d79d43ddc not found: ID does not exist" Oct 30 21:50:27.051387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.051359 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:50:27.097805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.097765 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-p9r22"] Oct 30 21:50:27.159668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.159593 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:50:27.164029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.164003 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-h5hrl"] Oct 30 21:50:27.430036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.429955 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" path="/var/lib/kubelet/pods/241d1ba1-dcc2-46df-a016-6d021f131514/volumes" Oct 30 21:50:27.430683 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:27.430656 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db2f944-3425-4d63-be41-4252daa4b028" path="/var/lib/kubelet/pods/2db2f944-3425-4d63-be41-4252daa4b028/volumes" Oct 30 21:50:28.039726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:28.037822 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:32.714883 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:32.714851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.342106 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:37.342072 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.537671 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:37.537620 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.738822 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:37.738786 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.135343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:42.135290 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.338561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:42.338527 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.529879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:42.529794 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.807816 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:50:45.807779 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:50:46.577086 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:46.577045 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.493492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:51.493463 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:56.244655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:56.244619 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.911142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:58.911047 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.295369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:50:59.295281 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.098514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:01.098468 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.792784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:03.792672 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:04.011580 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:04.011472 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.765728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:05.763099 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:10.315356 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:10.315255 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.761325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:14.761284 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.956449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:14.956397 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.531210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:15.531167 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.832798 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:15.832690 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:51:16.310484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:16.310451 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.356363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:19.356327 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.571100 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:19.571062 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.769861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:19.769805 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.389008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:23.388971 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.614336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:23.611725 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:24.882486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:24.882446 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:29.636828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:29.636780 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.675187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:33.675126 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.879018 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:33.878985 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.661272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.661240 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:51:34.661460 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.661441 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" podUID="86a7fb59-86f2-4ea7-adff-10a61b65d055" containerName="maas-api" containerID="cri-o://fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea" gracePeriod=30 Oct 30 21:51:34.767272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.765405 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:51:34.768204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.768167 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" containerID="cri-o://03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614" gracePeriod=30 Oct 30 21:51:34.772146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.771763 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:51:34.772146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.772001 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" containerID="cri-o://0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a" gracePeriod=30 Oct 30 21:51:34.778720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.778678 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:51:34.779011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.778978 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" containerID="cri-o://b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923" gracePeriod=30 Oct 30 21:51:34.790488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.790464 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:51:34.790650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.790631 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" containerID="cri-o://c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004" gracePeriod=30 Oct 30 21:51:34.828085 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:34.828048 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.006891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.006866 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.141978 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") pod \"86a7fb59-86f2-4ea7-adff-10a61b65d055\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.142160 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\") pod \"86a7fb59-86f2-4ea7-adff-10a61b65d055\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.142322 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\") pod \"86a7fb59-86f2-4ea7-adff-10a61b65d055\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.142436 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\" (UniqueName: \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\") pod \"86a7fb59-86f2-4ea7-adff-10a61b65d055\" (UID: \"86a7fb59-86f2-4ea7-adff-10a61b65d055\") " Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.144363 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca" pod "86a7fb59-86f2-4ea7-adff-10a61b65d055" (UID: "86a7fb59-86f2-4ea7-adff-10a61b65d055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.145054 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls" pod "86a7fb59-86f2-4ea7-adff-10a61b65d055" (UID: "86a7fb59-86f2-4ea7-adff-10a61b65d055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.145204 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.145227 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a7fb59-86f2-4ea7-adff-10a61b65d055-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.163736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.146445 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2" pod "86a7fb59-86f2-4ea7-adff-10a61b65d055" (UID: "86a7fb59-86f2-4ea7-adff-10a61b65d055"). InnerVolumeSpecName "kube-api-access-rgpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.168589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.168554 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs" pod "86a7fb59-86f2-4ea7-adff-10a61b65d055" (UID: "86a7fb59-86f2-4ea7-adff-10a61b65d055"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.176537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.176291 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.247220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.247025 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a7fb59-86f2-4ea7-adff-10a61b65d055-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.247220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.247065 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgpz2\" (UniqueName: \"kubernetes.io/projected/86a7fb59-86f2-4ea7-adff-10a61b65d055-kube-api-access-rgpz2\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.524489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.524395 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.820053 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.819971 4918 generic.go:356] "Generic (PLEG): container finished" podID="86a7fb59-86f2-4ea7-adff-10a61b65d055" containerID="fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea" exitCode=0 Oct 30 21:51:35.820933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.820907 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" event={"ID":"86a7fb59-86f2-4ea7-adff-10a61b65d055","Type":"ContainerDied","Data":"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea"} Oct 30 21:51:35.821115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.821099 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" event={"ID":"86a7fb59-86f2-4ea7-adff-10a61b65d055","Type":"ContainerDied","Data":"0179ce9ff68920647125dc6284d728ac657a9172e09b879bea84cb93c6c04ec3"} Oct 30 21:51:35.821224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.821213 4918 scope.go:122] "RemoveContainer" containerID="fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea" Oct 30 21:51:35.821489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.821477 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm" Oct 30 21:51:35.850668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.846987 4918 scope.go:122] "RemoveContainer" containerID="fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea" Oct 30 21:51:35.850668 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:35.848946 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea\": container with ID starting with fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea not found: ID does not exist" containerID="fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea" Oct 30 21:51:35.850668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.848989 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea"} err="failed to get container status \"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea\": rpc error: code = NotFound desc = could not find container \"fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea\": container with ID starting with fa3bcd98c75754b015ad600fb8fd602ccd55d2524116a63c5dda20546bc819ea not found: ID does not exist" Oct 30 21:51:35.858817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.858789 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.985728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.985398 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:51:35.996729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:35.991434 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-364e1518-595fb4bb87-gdxnm"] Oct 30 21:51:36.170737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:36.169973 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:36.170737 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:36.170031 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:36.306372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:36.306339 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.670639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:36.670512 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.858730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:36.857000 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:37.294726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:37.293121 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:37.433154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:37.433115 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a7fb59-86f2-4ea7-adff-10a61b65d055" path="/var/lib/kubelet/pods/86a7fb59-86f2-4ea7-adff-10a61b65d055/volumes" Oct 30 21:51:38.080890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:38.080854 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.267395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:38.267207 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.321648 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:38.321612 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:38.321990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:38.321953 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:38.676409 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:38.676377 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.144639 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.144608 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:51:40.239483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.230100 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:51:40.239483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.237836 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:51:40.274371 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274322 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274388 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274424 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274481 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274640 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274611 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274718 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274746 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.274931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274892 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.275033 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.274998 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\") pod \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\" (UID: \"163a23c1-1c65-40eb-a8c6-c63d2687aa93\") " Oct 30 21:51:40.276281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.276252 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.276959 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.276903 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.277232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.277209 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.277430 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.277403 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.280019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.279993 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.283006 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.282408 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:40.283006 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.282495 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.283296 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.283274 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "kube-api-access-jv6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.286410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.286379 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token" pod "163a23c1-1c65-40eb-a8c6-c63d2687aa93" (UID: "163a23c1-1c65-40eb-a8c6-c63d2687aa93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.292189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.292169 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:51:40.377779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.377752 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.378474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.378425 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.378644 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.378627 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.378881 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.378865 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.379416 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.379298 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.379867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.379824 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.379994 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.379377 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.380112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380097 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.380251 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380237 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.380391 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380374 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.380493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380480 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.380645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380630 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.380787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380771 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.380913 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380900 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.381013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381000 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.381115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381100 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.381225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381213 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.381344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381330 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.381448 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381435 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.381560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381547 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.381747 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381732 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.381979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381964 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.382099 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.382086 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.382235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.382222 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.382361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.382348 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.384535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.380508 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.381922 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.382662 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383043 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383109 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383138 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383168 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383222 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383471 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.384641 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.383981 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385217 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385301 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\") pod \"3009c681-1137-4f35-a754-9e535285bd45\" (UID: \"3009c681-1137-4f35-a754-9e535285bd45\") " Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385389 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\") pod \"403df095-8f5d-4087-a442-d6ea292c4b3e\" (UID: \"403df095-8f5d-4087-a442-d6ea292c4b3e\") " Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385430 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385461 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\") pod \"0786293a-a84a-4060-a771-2adbf773da4e\" (UID: \"0786293a-a84a-4060-a771-2adbf773da4e\") " Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.385667 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386251 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403df095-8f5d-4087-a442-d6ea292c4b3e-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386273 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386287 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386301 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386316 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386340 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386354 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3009c681-1137-4f35-a754-9e535285bd45-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386367 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386381 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386395 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386409 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386423 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386437 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6pg\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-kube-api-access-jv6pg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386451 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386464 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386478 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386491 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386505 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0786293a-a84a-4060-a771-2adbf773da4e-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386521 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386534 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386548 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386562 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386576 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163a23c1-1c65-40eb-a8c6-c63d2687aa93-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386846 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386945 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.386993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.386970 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3009c681-1137-4f35-a754-9e535285bd45-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.389742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.388264 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "kube-api-access-2chfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.389742 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.389674 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "kube-api-access-d557x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.389997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.389965 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "kube-api-access-slncp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.391310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.391284 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.391310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.391293 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:40.391473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.391307 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy" pod "0786293a-a84a-4060-a771-2adbf773da4e" (UID: "0786293a-a84a-4060-a771-2adbf773da4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.391473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.391445 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.392726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.392685 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:40.393019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.392819 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy" pod "3009c681-1137-4f35-a754-9e535285bd45" (UID: "3009c681-1137-4f35-a754-9e535285bd45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.393019 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.392932 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy" pod "403df095-8f5d-4087-a442-d6ea292c4b3e" (UID: "403df095-8f5d-4087-a442-d6ea292c4b3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:40.488907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488865 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.488907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488906 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488920 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488936 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3009c681-1137-4f35-a754-9e535285bd45-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488950 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488965 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0786293a-a84a-4060-a771-2adbf773da4e-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488978 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.488992 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chfq\" (UniqueName: \"kubernetes.io/projected/403df095-8f5d-4087-a442-d6ea292c4b3e-kube-api-access-2chfq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.489008 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slncp\" (UniqueName: \"kubernetes.io/projected/3009c681-1137-4f35-a754-9e535285bd45-kube-api-access-slncp\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.489022 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d557x\" (UniqueName: \"kubernetes.io/projected/0786293a-a84a-4060-a771-2adbf773da4e-kube-api-access-d557x\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.489150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.489036 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0786293a-a84a-4060-a771-2adbf773da4e-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.570511 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.570475 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.891901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.891855 4918 generic.go:356] "Generic (PLEG): container finished" podID="3009c681-1137-4f35-a754-9e535285bd45" containerID="c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004" exitCode=0 Oct 30 21:51:40.891901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.891895 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" Oct 30 21:51:40.892149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.891934 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" event={"ID":"3009c681-1137-4f35-a754-9e535285bd45","Type":"ContainerDied","Data":"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004"} Oct 30 21:51:40.892149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.891995 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" event={"ID":"3009c681-1137-4f35-a754-9e535285bd45","Type":"ContainerDied","Data":"b5a306a5e84df68ea4b46a8a94473ce17706a161a757aed30d42d5d1f8c17cd6"} Oct 30 21:51:40.892149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.892026 4918 scope.go:122] "RemoveContainer" containerID="c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004" Oct 30 21:51:40.893592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.893564 4918 generic.go:356] "Generic (PLEG): container finished" podID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerID="0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a" exitCode=0 Oct 30 21:51:40.893690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.893617 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" Oct 30 21:51:40.893690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.893620 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" event={"ID":"163a23c1-1c65-40eb-a8c6-c63d2687aa93","Type":"ContainerDied","Data":"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a"} Oct 30 21:51:40.893838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.893760 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" event={"ID":"163a23c1-1c65-40eb-a8c6-c63d2687aa93","Type":"ContainerDied","Data":"9c8bc1e5457b5f294036e7756ed3221bece790bd52dab564a8790dceef46811e"} Oct 30 21:51:40.896472 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.896448 4918 generic.go:356] "Generic (PLEG): container finished" podID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerID="03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614" exitCode=0 Oct 30 21:51:40.896598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.896492 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" Oct 30 21:51:40.896598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.896531 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" event={"ID":"403df095-8f5d-4087-a442-d6ea292c4b3e","Type":"ContainerDied","Data":"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614"} Oct 30 21:51:40.896598 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.896578 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2" event={"ID":"403df095-8f5d-4087-a442-d6ea292c4b3e","Type":"ContainerDied","Data":"b49703c7868e7cb3b759feac9120501ab3c0d6232577e710c8bbf86e72b14437"} Oct 30 21:51:40.898078 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.898055 4918 generic.go:356] "Generic (PLEG): container finished" podID="0786293a-a84a-4060-a771-2adbf773da4e" containerID="b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923" exitCode=0 Oct 30 21:51:40.898239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.898217 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" event={"ID":"0786293a-a84a-4060-a771-2adbf773da4e","Type":"ContainerDied","Data":"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923"} Oct 30 21:51:40.898313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.898253 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" event={"ID":"0786293a-a84a-4060-a771-2adbf773da4e","Type":"ContainerDied","Data":"acb6f9cde6e5409e655d5cbadfa6c9a6e5aeab4ab623ab51d0092f57890c7139"} Oct 30 21:51:40.898371 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.898317 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh" Oct 30 21:51:40.905259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.905061 4918 scope.go:122] "RemoveContainer" containerID="c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004" Oct 30 21:51:40.905402 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:40.905380 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004\": container with ID starting with c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004 not found: ID does not exist" containerID="c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004" Oct 30 21:51:40.905535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.905413 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004"} err="failed to get container status \"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004\": rpc error: code = NotFound desc = could not find container \"c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004\": container with ID starting with c9fc10562824ce216375ab431e8285a7f9d74352527850af9d58e7becf637004 not found: ID does not exist" Oct 30 21:51:40.905535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.905435 4918 scope.go:122] "RemoveContainer" containerID="0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a" Oct 30 21:51:40.917098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.917082 4918 scope.go:122] "RemoveContainer" containerID="0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a" Oct 30 21:51:40.917344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:40.917327 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a\": container with ID starting with 0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a not found: ID does not exist" containerID="0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a" Oct 30 21:51:40.917427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.917354 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a"} err="failed to get container status \"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a\": rpc error: code = NotFound desc = could not find container \"0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a\": container with ID starting with 0f4814e7aa614223e0517a8e9d14a2870c5eaff5727d00573ecb3f18d34afb3a not found: ID does not exist" Oct 30 21:51:40.917427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.917377 4918 scope.go:122] "RemoveContainer" containerID="03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614" Oct 30 21:51:40.926682 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.926658 4918 scope.go:122] "RemoveContainer" containerID="03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614" Oct 30 21:51:40.926973 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:40.926956 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614\": container with ID starting with 03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614 not found: ID does not exist" containerID="03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614" Oct 30 21:51:40.927063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.926979 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614"} err="failed to get container status \"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614\": rpc error: code = NotFound desc = could not find container \"03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614\": container with ID starting with 03039c768836ec4e594d694132d7d05b4e4ed17afc9b7869f592f2a5208f8614 not found: ID does not exist" Oct 30 21:51:40.927063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.926992 4918 scope.go:122] "RemoveContainer" containerID="b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923" Oct 30 21:51:40.939754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.939701 4918 scope.go:122] "RemoveContainer" containerID="b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923" Oct 30 21:51:40.940091 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:40.940063 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923\": container with ID starting with b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923 not found: ID does not exist" containerID="b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923" Oct 30 21:51:40.940202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:40.940098 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923"} err="failed to get container status \"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923\": rpc error: code = NotFound desc = could not find container \"b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923\": container with ID starting with b77c4b2bd1bc1ee60221f5a6ff5bc75f8ffe285a3dc5df3ece78c2f6dbdd0923 not found: ID does not exist" Oct 30 21:51:41.055744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.054054 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:51:41.060907 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.060842 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-sdzdh"] Oct 30 21:51:41.068084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.068053 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:51:41.068202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.068115 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:51:41.114923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.114887 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:51:41.115038 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.114958 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:51:41.162322 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.162292 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:51:41.169353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.169327 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rqrz2"] Oct 30 21:51:41.272961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.272912 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:51:41.277800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.277769 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-rx4s5"] Oct 30 21:51:41.388090 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.384793 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:51:41.392479 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.392444 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-lntm4"] Oct 30 21:51:41.430039 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.430007 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0786293a-a84a-4060-a771-2adbf773da4e" path="/var/lib/kubelet/pods/0786293a-a84a-4060-a771-2adbf773da4e/volumes" Oct 30 21:51:41.430614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.430593 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" path="/var/lib/kubelet/pods/163a23c1-1c65-40eb-a8c6-c63d2687aa93/volumes" Oct 30 21:51:41.431171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.431149 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3009c681-1137-4f35-a754-9e535285bd45" path="/var/lib/kubelet/pods/3009c681-1137-4f35-a754-9e535285bd45/volumes" Oct 30 21:51:41.432008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.431986 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" path="/var/lib/kubelet/pods/403df095-8f5d-4087-a442-d6ea292c4b3e/volumes" Oct 30 21:51:41.979557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:41.979520 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.176544 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:42.176497 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.532232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:42.532184 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.974625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:42.974592 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.167987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:43.167952 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.514943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:44.514908 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.696405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:44.696373 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.887789 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:44.887666 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.355912 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:45.355829 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.658833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:45.658797 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.806629 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:51:45.806593 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:51:45.845095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:45.845065 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.230860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:47.230817 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.280901 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.280864 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.497486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.496427 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.639763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.639701 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:51:48.640426 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640411 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640426 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640448 4918 cpu_manager.go:414] "containerMap: removing container" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640454 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640466 4918 cpu_manager.go:414] "containerMap: removing container" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640471 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640489 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640494 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" Oct 30 21:51:48.640499 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640503 4918 cpu_manager.go:414] "containerMap: removing container" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640508 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640525 4918 cpu_manager.go:414] "containerMap: removing container" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640530 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640540 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640545 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640558 4918 cpu_manager.go:414] "containerMap: removing container" podUID="86a7fb59-86f2-4ea7-adff-10a61b65d055" containerName="maas-api" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640562 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a7fb59-86f2-4ea7-adff-10a61b65d055" containerName="maas-api" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640655 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="241d1ba1-dcc2-46df-a016-6d021f131514" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640667 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="11939dde-6cbd-4750-9059-802b13696e34" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640677 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3009c681-1137-4f35-a754-9e535285bd45" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640691 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a7fb59-86f2-4ea7-adff-10a61b65d055" containerName="maas-api" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640699 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db2f944-3425-4d63-be41-4252daa4b028" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640730 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="403df095-8f5d-4087-a442-d6ea292c4b3e" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640739 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="163a23c1-1c65-40eb-a8c6-c63d2687aa93" containerName="istio-proxy" Oct 30 21:51:48.640857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.640749 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0786293a-a84a-4060-a771-2adbf773da4e" containerName="istio-proxy" Oct 30 21:51:48.646554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.646532 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.712730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.711159 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:51:48.741239 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.741202 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.741591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.741568 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwqz\" (UniqueName: \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.746764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.744102 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-27114d\"" Oct 30 21:51:48.746764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.744265 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-27114d-dockercfg-4dtt2\"" Oct 30 21:51:48.845935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.844916 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwqz\" (UniqueName: \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.845935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.845164 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.848650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.848624 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.861872 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.861843 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:51:48.884231 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.884204 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:51:48.888819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.888786 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:48.917261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.917120 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwqz\" (UniqueName: \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\") pod \"payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:48.958448 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.958410 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:48.958778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.958755 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6l7\" (UniqueName: \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:48.969902 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:48.969674 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:49.060514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.060485 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6l7\" (UniqueName: \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:49.060685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.060668 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:49.061444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.061423 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:49.128788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.123156 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6l7\" (UniqueName: \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\") pod \"payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:49.256010 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.255931 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:49.413537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.413489 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:51:49.417439 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:51:49.417400 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471f3883_ac0a_4781_90e5_77dd38f6fffe.slice/crio-1f27e1ab45b662f801e92e4ab2b23519a84b40bf2ea3f2a3851ef1171feabc9f WatchSource:0}: Error finding container 1f27e1ab45b662f801e92e4ab2b23519a84b40bf2ea3f2a3851ef1171feabc9f: Status 404 returned error can't find the container with id 1f27e1ab45b662f801e92e4ab2b23519a84b40bf2ea3f2a3851ef1171feabc9f Oct 30 21:51:49.646393 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:51:49.646349 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f0d01f_16c8_42ce_aa34_e0242d6bc53c.slice/crio-129d4f6d3a18d4a574d08a0533638942d74bf6e3302044553603f3115fd00720 WatchSource:0}: Error finding container 129d4f6d3a18d4a574d08a0533638942d74bf6e3302044553603f3115fd00720: Status 404 returned error can't find the container with id 129d4f6d3a18d4a574d08a0533638942d74bf6e3302044553603f3115fd00720 Oct 30 21:51:49.646527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:49.646468 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:51:50.042495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.042391 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" event={"ID":"471f3883-ac0a-4781-90e5-77dd38f6fffe","Type":"ContainerStarted","Data":"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294"} Oct 30 21:51:50.042495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.042458 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" event={"ID":"471f3883-ac0a-4781-90e5-77dd38f6fffe","Type":"ContainerStarted","Data":"1f27e1ab45b662f801e92e4ab2b23519a84b40bf2ea3f2a3851ef1171feabc9f"} Oct 30 21:51:50.042495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.042490 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:51:50.043963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.043937 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" event={"ID":"36f0d01f-16c8-42ce-aa34-e0242d6bc53c","Type":"ContainerStarted","Data":"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4"} Oct 30 21:51:50.044076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.043978 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" event={"ID":"36f0d01f-16c8-42ce-aa34-e0242d6bc53c","Type":"ContainerStarted","Data":"129d4f6d3a18d4a574d08a0533638942d74bf6e3302044553603f3115fd00720"} Oct 30 21:51:50.044076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.044057 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:51:50.155047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.154996 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" podStartSLOduration=2.154982564 podStartE2EDuration="2.154982564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:50.152213129 +0000 UTC m=+2290.845695365" watchObservedRunningTime="2026-10-30 21:51:50.154982564 +0000 UTC m=+2290.848464799" Oct 30 21:51:50.274190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:50.273909 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" podStartSLOduration=2.273887529 podStartE2EDuration="2.273887529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:50.26229763 +0000 UTC m=+2290.955779877" watchObservedRunningTime="2026-10-30 21:51:50.273887529 +0000 UTC m=+2290.967369766" Oct 30 21:51:51.936736 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:51.936263 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.132274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:52.132244 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.558866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:52.558829 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.165863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:53.165796 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.922467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:53.922422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.052024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:57.051982 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.272764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:57.272672 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.460119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:57.460087 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.060456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:58.060402 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.774813 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:58.774757 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.519638 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:51:59.519604 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.558882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:00.558847 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.759959 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:00.759917 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.952728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:00.952149 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:52:00.966731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:00.963092 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.010529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.010498 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:52:01.052217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.052189 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-737a64\"" Oct 30 21:52:01.052217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.052220 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-737a64-dockercfg-wlql5\"" Oct 30 21:52:01.055875 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.055854 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:52:01.056008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.055933 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:52:01.103630 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.103599 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:52:01.108685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.108658 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.117677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.117648 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8zh\" (UniqueName: \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.117886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.117863 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.145928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.145897 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:52:01.220446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.220355 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8zh\" (UniqueName: \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.220944 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.220918 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.221175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.221156 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.221316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.221301 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95m7\" (UniqueName: \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.226581 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.226554 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.284469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.284436 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8zh\" (UniqueName: \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\") pod \"payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.313189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.313159 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:01.329147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.329113 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.329510 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.329357 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95m7\" (UniqueName: \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.332374 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.332353 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.434007 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.433935 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95m7\" (UniqueName: \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\") pod \"payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.436538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.436494 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:01.803764 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:01.803736 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec34a0f9_2604_4ce0_897b_786f983d1fad.slice/crio-c9ba5547121be600cc66f6e53959a16ed56fd072cde0df03efcc7fc896048a0e WatchSource:0}: Error finding container c9ba5547121be600cc66f6e53959a16ed56fd072cde0df03efcc7fc896048a0e: Status 404 returned error can't find the container with id c9ba5547121be600cc66f6e53959a16ed56fd072cde0df03efcc7fc896048a0e Oct 30 21:52:01.804208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.804106 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:52:01.851083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:01.851056 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:52:01.853378 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:01.853352 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f5b08c_08af_4128_a346_2880773b5047.slice/crio-0375f1e9b03672a059fd200d8dc92af43da695fc477b99ca2bbdbcca8033a64f WatchSource:0}: Error finding container 0375f1e9b03672a059fd200d8dc92af43da695fc477b99ca2bbdbcca8033a64f: Status 404 returned error can't find the container with id 0375f1e9b03672a059fd200d8dc92af43da695fc477b99ca2bbdbcca8033a64f Oct 30 21:52:02.195792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.195750 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" event={"ID":"84f5b08c-08af-4128-a346-2880773b5047","Type":"ContainerStarted","Data":"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555"} Oct 30 21:52:02.195975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.195821 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" event={"ID":"84f5b08c-08af-4128-a346-2880773b5047","Type":"ContainerStarted","Data":"0375f1e9b03672a059fd200d8dc92af43da695fc477b99ca2bbdbcca8033a64f"} Oct 30 21:52:02.195975 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.195934 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:02.197431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.197394 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" event={"ID":"ec34a0f9-2604-4ce0-897b-786f983d1fad","Type":"ContainerStarted","Data":"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c"} Oct 30 21:52:02.197545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.197453 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" event={"ID":"ec34a0f9-2604-4ce0-897b-786f983d1fad","Type":"ContainerStarted","Data":"c9ba5547121be600cc66f6e53959a16ed56fd072cde0df03efcc7fc896048a0e"} Oct 30 21:52:02.197545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.197484 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:02.312500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.312450 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" podStartSLOduration=1.312437424 podStartE2EDuration="1.312437424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:02.308920258 +0000 UTC m=+2303.002402488" watchObservedRunningTime="2026-10-30 21:52:02.312437424 +0000 UTC m=+2303.005919659" Oct 30 21:52:02.338891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.324869 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:02.435729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:02.425451 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" podStartSLOduration=2.425430005 podStartE2EDuration="2.425430005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:02.421068152 +0000 UTC m=+2303.114550389" watchObservedRunningTime="2026-10-30 21:52:02.425430005 +0000 UTC m=+2303.118912241" Oct 30 21:52:03.403923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:03.403886 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:03.608547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:03.608492 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:05.505555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.505518 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:52:05.506062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.505790 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" containerID="cri-o://88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2" gracePeriod=30 Oct 30 21:52:05.552519 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.552492 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:52:05.552745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.552722 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" containerID="cri-o://1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5" gracePeriod=30 Oct 30 21:52:05.556645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.556623 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:52:05.556810 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.556791 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" containerID="cri-o://dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb" gracePeriod=30 Oct 30 21:52:05.599567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.599543 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:52:05.599760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:05.599737 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" containerID="cri-o://261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76" gracePeriod=30 Oct 30 21:52:06.736438 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:06.736401 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:06.956696 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:06.956632 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.101091 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.101001 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:52:07.112055 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.112010 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.132651 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.132617 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfzz\" (UniqueName: \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.132905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.132878 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.161456 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.161412 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:52:07.216223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.209027 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-2aa268\"" Oct 30 21:52:07.216223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.209190 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-2aa268-dockercfg-w4fh2\"" Oct 30 21:52:07.239845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.239807 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.240160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.240141 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfzz\" (UniqueName: \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.241620 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.241585 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.285270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.285230 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:52:07.296045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.292249 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 21:52:07.297568 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.292659 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.305079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.304949 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.311694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.309102 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfzz\" (UniqueName: \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\") pod \"payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.343639 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.343858 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8c66\" (UniqueName: \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.343944 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.343964 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.344121 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.344205 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8vz\" (UniqueName: \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.346768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.344238 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.355336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.355270 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-40bb78-serving-cert\"" Oct 30 21:52:07.364299 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.364262 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 21:52:07.439383 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.439115 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445118 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8c66\" (UniqueName: \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445175 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445348 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8vz\" (UniqueName: \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.445560 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.449738 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.450903 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:07.451183 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-serving-cert: secret "maas-api-e2e-shared-a-w5-40bb78-serving-cert" not found Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:07.451238 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls podName:6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:07.951220426 +0000 UTC m=+2308.644702644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls") pod "maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" (UID: "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1") : secret "maas-api-e2e-shared-a-w5-40bb78-serving-cert" not found Oct 30 21:52:07.453387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.452901 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.500729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.500142 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.534651 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.528621 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8vz\" (UniqueName: \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.545692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.545639 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8c66\" (UniqueName: \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\") pod \"payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.683789 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.676447 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:07.697052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.696972 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.881939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.881903 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.939948 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:07.939906 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28f5ed2_9120_4340_9eb2_b282ee98d733.slice/crio-31f0f64b4b69ffd561ab257c0cab6e345c02ec64273c0c70437f6b1ebe67e25c WatchSource:0}: Error finding container 31f0f64b4b69ffd561ab257c0cab6e345c02ec64273c0c70437f6b1ebe67e25c: Status 404 returned error can't find the container with id 31f0f64b4b69ffd561ab257c0cab6e345c02ec64273c0c70437f6b1ebe67e25c Oct 30 21:52:07.940259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.940233 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:52:07.953628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.953601 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:07.955990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:07.955970 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:08.008859 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.008837 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:08.102978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.101025 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:52:08.130491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.130460 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:08.130594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.130516 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:08.315467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.315378 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" event={"ID":"c28f5ed2-9120-4340-9eb2-b282ee98d733","Type":"ContainerStarted","Data":"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163"} Oct 30 21:52:08.315467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.315441 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" event={"ID":"c28f5ed2-9120-4340-9eb2-b282ee98d733","Type":"ContainerStarted","Data":"31f0f64b4b69ffd561ab257c0cab6e345c02ec64273c0c70437f6b1ebe67e25c"} Oct 30 21:52:08.315467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.315465 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:08.317308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.317269 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" event={"ID":"0bde51c2-d7cb-47c0-be0b-018e87d3449a","Type":"ContainerStarted","Data":"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354"} Oct 30 21:52:08.317455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.317319 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" event={"ID":"0bde51c2-d7cb-47c0-be0b-018e87d3449a","Type":"ContainerStarted","Data":"cdd87d070cea6d2538e88027ffb196254b27287a16d943851feb927dbcee9fe5"} Oct 30 21:52:08.317532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.317478 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:08.383845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.383806 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 21:52:08.384158 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:08.384126 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6684ee9f_a76a_4cf0_8c4f_f9539c6ee0d1.slice/crio-6b00d864e96c63b561a41edeefd94ebefc02bc90ed60cb5e287b0e74915a915e WatchSource:0}: Error finding container 6b00d864e96c63b561a41edeefd94ebefc02bc90ed60cb5e287b0e74915a915e: Status 404 returned error can't find the container with id 6b00d864e96c63b561a41edeefd94ebefc02bc90ed60cb5e287b0e74915a915e Oct 30 21:52:08.385660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.385640 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:52:08.429212 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.429167 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" podStartSLOduration=1.429154825 podStartE2EDuration="1.429154825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:08.427725677 +0000 UTC m=+2309.121207910" watchObservedRunningTime="2026-10-30 21:52:08.429154825 +0000 UTC m=+2309.122637060" Oct 30 21:52:08.542288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.539225 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" podStartSLOduration=1.5392082280000001 podStartE2EDuration="1.539208228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:08.538812913 +0000 UTC m=+2309.232295150" watchObservedRunningTime="2026-10-30 21:52:08.539208228 +0000 UTC m=+2309.232690465" Oct 30 21:52:08.939886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.939845 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:08.940317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:08.939915 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:09.327014 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:09.326983 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" event={"ID":"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1","Type":"ContainerStarted","Data":"6b00d864e96c63b561a41edeefd94ebefc02bc90ed60cb5e287b0e74915a915e"} Oct 30 21:52:10.351784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.347773 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" event={"ID":"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1","Type":"ContainerStarted","Data":"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4"} Oct 30 21:52:10.356570 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.356531 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:10.475081 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.475013 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" podStartSLOduration=2.565123609 podStartE2EDuration="3.474994242s" totalImagesPullingTime="909.870633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:10.471881341 +0000 UTC m=+2311.165363578" watchObservedRunningTime="2026-10-30 21:52:10.474994242 +0000 UTC m=+2311.168476478" Oct 30 21:52:10.874187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.874163 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:52:10.904956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.904886 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.904958 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905001 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905089 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905079 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905211 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905326 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905367 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905484 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905626 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\") pod \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\" (UID: \"3b4f1503-cd63-49b5-81f8-58c8290d79ed\") " Oct 30 21:52:10.905853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.905815 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.906352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.906325 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.906352 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.906342 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.907152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.906500 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:10.907152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.907123 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.908121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.908097 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.910756 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.910721 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "kube-api-access-2clmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:10.911866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.911832 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:10.917058 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.917036 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.918647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:10.918293 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo" pod "3b4f1503-cd63-49b5-81f8-58c8290d79ed" (UID: "3b4f1503-cd63-49b5-81f8-58c8290d79ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:11.008062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008035 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:52:11.008216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008188 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clmz\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-kube-api-access-2clmz\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008347 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008520 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008545 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008559 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008572 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008586 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4f1503-cd63-49b5-81f8-58c8290d79ed-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.008674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.008599 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4f1503-cd63-49b5-81f8-58c8290d79ed-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.010485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.010413 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:52:11.017263 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.017241 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111214 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111285 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111328 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111364 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111394 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111493 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111530 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111592 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111699 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111818 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111913 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.111958 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.112194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.112015 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.113207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113112 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113320 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113226 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113320 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113270 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.113441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113360 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.113441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113392 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113437 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113464 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.113552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113521 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.113764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113726 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113781 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.113905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113817 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\") pod \"ac21b833-45e1-43b7-a112-0e9e6cad9332\" (UID: \"ac21b833-45e1-43b7-a112-0e9e6cad9332\") " Oct 30 21:52:11.113905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113856 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\") pod \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\" (UID: \"ce138bce-f56c-4abb-9b2e-efbdb3b342ed\") " Oct 30 21:52:11.114198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113955 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.114198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.113982 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\") pod \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\" (UID: \"0e09117f-7e66-4e32-9b3d-5e4058fde4fc\") " Oct 30 21:52:11.116063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.115667 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.116063 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.115970 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.116344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.116310 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.116468 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.116344 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:11.116682 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.116609 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.116819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.116685 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:11.116883 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.116869 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.117376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.117318 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:11.117768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.117603 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.117911 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.117890 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.118000 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.117915 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.118495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.118125 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.118495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.118138 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.118495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.118339 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.118495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.118477 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.121759 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.121860 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.121977 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122048 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122172 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "kube-api-access-bnv7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.122224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122198 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "kube-api-access-xxvxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.122599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122295 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "kube-api-access-c767h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.122599 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122357 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.122756 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122737 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:11.122846 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122831 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo" pod "ac21b833-45e1-43b7-a112-0e9e6cad9332" (UID: "ac21b833-45e1-43b7-a112-0e9e6cad9332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:11.122900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.122837 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy" pod "0e09117f-7e66-4e32-9b3d-5e4058fde4fc" (UID: "0e09117f-7e66-4e32-9b3d-5e4058fde4fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:11.123359 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.123342 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token" pod "ce138bce-f56c-4abb-9b2e-efbdb3b342ed" (UID: "ce138bce-f56c-4abb-9b2e-efbdb3b342ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216542 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216572 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216582 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216592 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216602 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216612 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216615 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216620 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac21b833-45e1-43b7-a112-0e9e6cad9332-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216628 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216636 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216645 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216653 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216661 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216670 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216679 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216688 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvxf\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-kube-api-access-xxvxf\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216696 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216734 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac21b833-45e1-43b7-a112-0e9e6cad9332-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216744 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv7k\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-kube-api-access-bnv7k\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216753 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216762 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216770 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216780 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce138bce-f56c-4abb-9b2e-efbdb3b342ed-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216789 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216798 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216806 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac21b833-45e1-43b7-a112-0e9e6cad9332-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216814 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c767h\" (UniqueName: \"kubernetes.io/projected/ac21b833-45e1-43b7-a112-0e9e6cad9332-kube-api-access-c767h\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.216982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.216823 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e09117f-7e66-4e32-9b3d-5e4058fde4fc-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:11.368494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.368457 4918 generic.go:356] "Generic (PLEG): container finished" podID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerID="1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5" exitCode=0 Oct 30 21:52:11.369049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.368505 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" Oct 30 21:52:11.369049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.368549 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerDied","Data":"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5"} Oct 30 21:52:11.369049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.368615 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f" event={"ID":"0e09117f-7e66-4e32-9b3d-5e4058fde4fc","Type":"ContainerDied","Data":"379aac9bf42004f43c1bd56503387486b8e8f9b2d154781298b5d492e618d443"} Oct 30 21:52:11.369049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.368642 4918 scope.go:122] "RemoveContainer" containerID="1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5" Oct 30 21:52:11.371103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.370641 4918 generic.go:356] "Generic (PLEG): container finished" podID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerID="dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb" exitCode=0 Oct 30 21:52:11.371103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.370769 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" Oct 30 21:52:11.371103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.370765 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" event={"ID":"ce138bce-f56c-4abb-9b2e-efbdb3b342ed","Type":"ContainerDied","Data":"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb"} Oct 30 21:52:11.371103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.370954 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk" event={"ID":"ce138bce-f56c-4abb-9b2e-efbdb3b342ed","Type":"ContainerDied","Data":"51074560a2d40ff5d05cd82ccd5db33a7748c7b499a049ea465bfd44b640fec4"} Oct 30 21:52:11.373614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.373177 4918 generic.go:356] "Generic (PLEG): container finished" podID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerID="261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76" exitCode=0 Oct 30 21:52:11.373614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.373231 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" Oct 30 21:52:11.373614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.373337 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" event={"ID":"ac21b833-45e1-43b7-a112-0e9e6cad9332","Type":"ContainerDied","Data":"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76"} Oct 30 21:52:11.373614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.373382 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2" event={"ID":"ac21b833-45e1-43b7-a112-0e9e6cad9332","Type":"ContainerDied","Data":"485cd6f61cbbd07fadd5536a8ab6955ed5b6ea1ded8e6b44a9045559f4deab32"} Oct 30 21:52:11.387817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.387060 4918 scope.go:122] "RemoveContainer" containerID="2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b" Oct 30 21:52:11.388857 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.388835 4918 generic.go:356] "Generic (PLEG): container finished" podID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerID="88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2" exitCode=0 Oct 30 21:52:11.389036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.389006 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" event={"ID":"3b4f1503-cd63-49b5-81f8-58c8290d79ed","Type":"ContainerDied","Data":"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2"} Oct 30 21:52:11.389149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.389071 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" event={"ID":"3b4f1503-cd63-49b5-81f8-58c8290d79ed","Type":"ContainerDied","Data":"ff8eeca9e3b4d9e4053f8ba116df82cac0db628cbcd9bab51c7331e29844823e"} Oct 30 21:52:11.389284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.389262 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg" Oct 30 21:52:11.406740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.406720 4918 scope.go:122] "RemoveContainer" containerID="1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5" Oct 30 21:52:11.407185 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:11.407161 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5\": container with ID starting with 1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5 not found: ID does not exist" containerID="1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5" Oct 30 21:52:11.407307 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.407193 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5"} err="failed to get container status \"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5\": rpc error: code = NotFound desc = could not find container \"1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5\": container with ID starting with 1c9830d4a2dc58a656f52bf4bec13ac7b3a4eab5807e3019efd068d00e14d6d5 not found: ID does not exist" Oct 30 21:52:11.407307 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.407210 4918 scope.go:122] "RemoveContainer" containerID="2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b" Oct 30 21:52:11.407518 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:11.407499 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b\": container with ID starting with 2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b not found: ID does not exist" containerID="2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b" Oct 30 21:52:11.407583 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.407523 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b"} err="failed to get container status \"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b\": rpc error: code = NotFound desc = could not find container \"2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b\": container with ID starting with 2b12a9a8f77d67dffc997aadec8cca766db5da5704f0765fba3d74581c73d79b not found: ID does not exist" Oct 30 21:52:11.407583 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.407537 4918 scope.go:122] "RemoveContainer" containerID="dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb" Oct 30 21:52:11.418185 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.418168 4918 scope.go:122] "RemoveContainer" containerID="dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb" Oct 30 21:52:11.418474 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:11.418453 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb\": container with ID starting with dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb not found: ID does not exist" containerID="dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb" Oct 30 21:52:11.418558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.418479 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb"} err="failed to get container status \"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb\": rpc error: code = NotFound desc = could not find container \"dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb\": container with ID starting with dbc81a15a78747eaed46f4f47f05a0b7e70f371afa7daa48232348902be546eb not found: ID does not exist" Oct 30 21:52:11.418558 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.418496 4918 scope.go:122] "RemoveContainer" containerID="261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76" Oct 30 21:52:11.431595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.431566 4918 scope.go:122] "RemoveContainer" containerID="261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76" Oct 30 21:52:11.431888 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:11.431860 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76\": container with ID starting with 261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76 not found: ID does not exist" containerID="261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76" Oct 30 21:52:11.432056 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.432026 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76"} err="failed to get container status \"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76\": rpc error: code = NotFound desc = could not find container \"261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76\": container with ID starting with 261b22262c96f8e48111d323ae73203139d48a2532145be2557279bac2744c76 not found: ID does not exist" Oct 30 21:52:11.432151 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.432059 4918 scope.go:122] "RemoveContainer" containerID="88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2" Oct 30 21:52:11.444593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.444577 4918 scope.go:122] "RemoveContainer" containerID="88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2" Oct 30 21:52:11.444900 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:11.444882 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2\": container with ID starting with 88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2 not found: ID does not exist" containerID="88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2" Oct 30 21:52:11.444973 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.444907 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2"} err="failed to get container status \"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2\": rpc error: code = NotFound desc = could not find container \"88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2\": container with ID starting with 88a67d38ec9949f0d3287f52925ba7d5bd82cf4eb1e8a636c9afe5deefd80fc2 not found: ID does not exist" Oct 30 21:52:11.541946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.541913 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:52:11.546474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.546453 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mlqh2"] Oct 30 21:52:11.655270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.655225 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:52:11.660436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.660401 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mtzmk"] Oct 30 21:52:11.761600 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.761509 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:52:11.766720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.766685 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-k5k5f"] Oct 30 21:52:11.878678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.878644 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:52:11.885166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:11.885122 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-rlccg"] Oct 30 21:52:12.707782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:12.702360 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:13.211537 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.211508 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:52:13.211850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.211831 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:52:13.433858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.433816 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" path="/var/lib/kubelet/pods/0e09117f-7e66-4e32-9b3d-5e4058fde4fc/volumes" Oct 30 21:52:13.437763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.437720 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" path="/var/lib/kubelet/pods/3b4f1503-cd63-49b5-81f8-58c8290d79ed/volumes" Oct 30 21:52:13.438389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.438371 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" path="/var/lib/kubelet/pods/ac21b833-45e1-43b7-a112-0e9e6cad9332/volumes" Oct 30 21:52:13.439312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:13.439252 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" path="/var/lib/kubelet/pods/ce138bce-f56c-4abb-9b2e-efbdb3b342ed/volumes" Oct 30 21:52:15.805168 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:15.805132 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:52:16.396694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:16.396666 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 21:52:17.447434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:17.447395 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:19.330444 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:19.330412 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:52:19.330923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:19.330479 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:52:21.613732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:21.611891 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.108167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.108077 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.171782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.171741 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:52:22.171996 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.171958 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" containerID="cri-o://1d6d4478cf8db831fd4afe81f356b1b9bbe3105b69aac814c14f01ba829ef67d" gracePeriod=30 Oct 30 21:52:22.220042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.220007 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:52:22.220319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.220267 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" containerID="cri-o://6fa42ad8fefc5d49c89519004dbcf8f4696c4805a2cf507a40fb22cf256ca68a" gracePeriod=30 Oct 30 21:52:22.226525 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.226500 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:52:22.226694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.226664 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" containerID="cri-o://f642ae2debb0053b4510f1fa96b8ae861a7bbabe4b352f98861870b7ac9d3b8d" gracePeriod=30 Oct 30 21:52:22.263896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.263870 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:52:22.264065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:22.264040 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" containerID="cri-o://c6ebaff5ee6c22f54e6af10be112f7e9faea87f69d6eff913817e80d34f69dd4" gracePeriod=30 Oct 30 21:52:24.828032 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:24.827998 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:24.828538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:24.828047 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:26.109122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:26.109090 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:26.109506 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:26.109177 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:26.727768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:26.727737 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:27.477530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.477154 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:27.477530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.477212 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:52:27.492771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.492648 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:52:27.493001 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.492924 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:52:27.552027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.551991 4918 generic.go:356] "Generic (PLEG): container finished" podID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerID="c6ebaff5ee6c22f54e6af10be112f7e9faea87f69d6eff913817e80d34f69dd4" exitCode=0 Oct 30 21:52:27.552147 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.552131 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" event={"ID":"04ab9c7b-871b-46a9-be09-022d3fe9db5f","Type":"ContainerDied","Data":"c6ebaff5ee6c22f54e6af10be112f7e9faea87f69d6eff913817e80d34f69dd4"} Oct 30 21:52:27.554521 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.554494 4918 generic.go:356] "Generic (PLEG): container finished" podID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerID="1d6d4478cf8db831fd4afe81f356b1b9bbe3105b69aac814c14f01ba829ef67d" exitCode=0 Oct 30 21:52:27.554602 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.554577 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" event={"ID":"99c9b5fd-8833-4836-829e-abbfaaeef9e4","Type":"ContainerDied","Data":"1d6d4478cf8db831fd4afe81f356b1b9bbe3105b69aac814c14f01ba829ef67d"} Oct 30 21:52:27.554649 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.554627 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" event={"ID":"99c9b5fd-8833-4836-829e-abbfaaeef9e4","Type":"ContainerDied","Data":"293d22bc6ec7c13c4c9a8cfce9e0a5ec06599f91c87b7b6d96fd34d3c74e7c88"} Oct 30 21:52:27.554688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.554649 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="293d22bc6ec7c13c4c9a8cfce9e0a5ec06599f91c87b7b6d96fd34d3c74e7c88" Oct 30 21:52:27.557003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.556979 4918 generic.go:356] "Generic (PLEG): container finished" podID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerID="6fa42ad8fefc5d49c89519004dbcf8f4696c4805a2cf507a40fb22cf256ca68a" exitCode=0 Oct 30 21:52:27.557136 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.557117 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" event={"ID":"55f02365-e1e7-48ac-85c2-b39f0dca13f2","Type":"ContainerDied","Data":"6fa42ad8fefc5d49c89519004dbcf8f4696c4805a2cf507a40fb22cf256ca68a"} Oct 30 21:52:27.559848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.559795 4918 generic.go:356] "Generic (PLEG): container finished" podID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerID="f642ae2debb0053b4510f1fa96b8ae861a7bbabe4b352f98861870b7ac9d3b8d" exitCode=0 Oct 30 21:52:27.559848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.559853 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" event={"ID":"ad891f0b-3f80-471f-b1e2-5c53bf70fe09","Type":"ContainerDied","Data":"f642ae2debb0053b4510f1fa96b8ae861a7bbabe4b352f98861870b7ac9d3b8d"} Oct 30 21:52:27.561103 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.561079 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643067 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643174 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643207 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643308 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643352 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643473 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643538 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643592 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.644559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.643672 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\") pod \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\" (UID: \"99c9b5fd-8833-4836-829e-abbfaaeef9e4\") " Oct 30 21:52:27.645461 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.645357 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.646927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.646649 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.646927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.646885 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.647250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.647203 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.648044 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.648015 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:27.654624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.654600 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "kube-api-access-znqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.659395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.657074 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:27.659395 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.657502 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.661070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.661045 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy" pod "99c9b5fd-8833-4836-829e-abbfaaeef9e4" (UID: "99c9b5fd-8833-4836-829e-abbfaaeef9e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.697766 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.697527 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:52:27.715210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.715134 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:52:27.719260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.719237 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:52:27.745591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745533 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.745591 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745589 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745612 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745634 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745671 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745690 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745724 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745759 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.745788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745783 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745840 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745862 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745880 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745919 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745943 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.745986 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746009 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746032 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746074 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746095 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746130 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746174 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746196 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746235 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746262 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\") pod \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\" (UID: \"ad891f0b-3f80-471f-b1e2-5c53bf70fe09\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746329 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\") pod \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\" (UID: \"04ab9c7b-871b-46a9-be09-022d3fe9db5f\") " Oct 30 21:52:27.746826 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.746375 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\") pod \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\" (UID: \"55f02365-e1e7-48ac-85c2-b39f0dca13f2\") " Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747037 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747153 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747179 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747195 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747209 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747223 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747224 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747236 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747249 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znqn2\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-kube-api-access-znqn2\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747263 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.747963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747372 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.747963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.747878 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.749194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.749173 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.749594 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.749430 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9b5fd-8833-4836-829e-abbfaaeef9e4-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.749698 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.749649 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.749800 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.749692 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:27.750095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750074 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.750186 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750113 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.750290 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750272 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.750457 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750440 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.750535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750492 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:27.751592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.750907 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.751592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.751271 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.751592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.751290 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.751592 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.751415 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.753743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.753687 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:27.755221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.755200 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:27.755501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.755482 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.756065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756025 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.756065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756039 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:27.756237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756068 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "kube-api-access-xbcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.756237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756086 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.756237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756100 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7" pod "55f02365-e1e7-48ac-85c2-b39f0dca13f2" (UID: "55f02365-e1e7-48ac-85c2-b39f0dca13f2"). InnerVolumeSpecName "kube-api-access-vjlw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.756237 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.756183 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.757062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.757040 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy" pod "04ab9c7b-871b-46a9-be09-022d3fe9db5f" (UID: "04ab9c7b-871b-46a9-be09-022d3fe9db5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.757242 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.757223 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:27.757433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.757416 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66" pod "ad891f0b-3f80-471f-b1e2-5c53bf70fe09" (UID: "ad891f0b-3f80-471f-b1e2-5c53bf70fe09"). InnerVolumeSpecName "kube-api-access-4xf66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851630 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851660 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851671 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcxn\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-kube-api-access-xbcxn\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851681 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851689 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.851665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851697 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851745 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851756 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851765 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851773 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851782 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851791 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xf66\" (UniqueName: \"kubernetes.io/projected/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-kube-api-access-4xf66\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851799 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851808 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851816 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851826 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851833 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851841 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851850 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851858 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851867 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851874 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f02365-e1e7-48ac-85c2-b39f0dca13f2-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851884 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjlw7\" (UniqueName: \"kubernetes.io/projected/55f02365-e1e7-48ac-85c2-b39f0dca13f2-kube-api-access-vjlw7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851892 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851901 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ab9c7b-871b-46a9-be09-022d3fe9db5f-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851909 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad891f0b-3f80-471f-b1e2-5c53bf70fe09-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:27.852469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:27.851918 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f02365-e1e7-48ac-85c2-b39f0dca13f2-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:28.572732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.570529 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" Oct 30 21:52:28.572732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.570560 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7" event={"ID":"04ab9c7b-871b-46a9-be09-022d3fe9db5f","Type":"ContainerDied","Data":"a4d3ce2f1a4c89ef5b1635007defa981567ca7806d52e4931d93da8202552213"} Oct 30 21:52:28.572732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.570672 4918 scope.go:122] "RemoveContainer" containerID="c6ebaff5ee6c22f54e6af10be112f7e9faea87f69d6eff913817e80d34f69dd4" Oct 30 21:52:28.573597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.572748 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" event={"ID":"55f02365-e1e7-48ac-85c2-b39f0dca13f2","Type":"ContainerDied","Data":"f0332a085d980d036f028f03280bc785e17f342d3b170621eb41b3c0fdb36a51"} Oct 30 21:52:28.573597 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.572776 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9" Oct 30 21:52:28.574538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.574510 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" Oct 30 21:52:28.574655 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.574565 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv" event={"ID":"ad891f0b-3f80-471f-b1e2-5c53bf70fe09","Type":"ContainerDied","Data":"9a25a0881422c2904776dac136043bcf51f22aa6e97e0ee0b93d7609d31f154a"} Oct 30 21:52:28.574827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.574812 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz" Oct 30 21:52:28.585546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.585483 4918 scope.go:122] "RemoveContainer" containerID="6fa42ad8fefc5d49c89519004dbcf8f4696c4805a2cf507a40fb22cf256ca68a" Oct 30 21:52:28.597990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.597971 4918 scope.go:122] "RemoveContainer" containerID="f642ae2debb0053b4510f1fa96b8ae861a7bbabe4b352f98861870b7ac9d3b8d" Oct 30 21:52:28.678125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.678098 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.777211 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.777177 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:52:28.823365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.823280 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-7m6kv"] Oct 30 21:52:28.884696 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.884666 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:52:28.888699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.888677 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-d2wj7"] Oct 30 21:52:28.992929 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.992899 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:52:28.996634 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:28.996610 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-8hgwz"] Oct 30 21:52:29.100900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.100832 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:52:29.105027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.105005 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-gb2c9"] Oct 30 21:52:29.429689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.429660 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" path="/var/lib/kubelet/pods/04ab9c7b-871b-46a9-be09-022d3fe9db5f/volumes" Oct 30 21:52:29.430267 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.430249 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" path="/var/lib/kubelet/pods/55f02365-e1e7-48ac-85c2-b39f0dca13f2/volumes" Oct 30 21:52:29.430698 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.430684 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" path="/var/lib/kubelet/pods/99c9b5fd-8833-4836-829e-abbfaaeef9e4/volumes" Oct 30 21:52:29.431494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:29.431478 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" path="/var/lib/kubelet/pods/ad891f0b-3f80-471f-b1e2-5c53bf70fe09/volumes" Oct 30 21:52:31.044411 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.044372 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:52:31.045122 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045102 4918 cpu_manager.go:414] "containerMap: removing container" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045124 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045135 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045140 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045151 4918 cpu_manager.go:414] "containerMap: removing container" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045155 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045164 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045168 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045176 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045180 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045199 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045204 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045210 4918 cpu_manager.go:414] "containerMap: removing container" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045215 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045241 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045245 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045261 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045265 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045361 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4f1503-cd63-49b5-81f8-58c8290d79ed" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045372 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac21b833-45e1-43b7-a112-0e9e6cad9332" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045380 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ab9c7b-871b-46a9-be09-022d3fe9db5f" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045387 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045393 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c9b5fd-8833-4836-829e-abbfaaeef9e4" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045400 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce138bce-f56c-4abb-9b2e-efbdb3b342ed" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045406 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e09117f-7e66-4e32-9b3d-5e4058fde4fc" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045415 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad891f0b-3f80-471f-b1e2-5c53bf70fe09" containerName="istio-proxy" Oct 30 21:52:31.045619 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.045419 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f02365-e1e7-48ac-85c2-b39f0dca13f2" containerName="istio-proxy" Oct 30 21:52:31.050697 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.050677 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.091248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.091220 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.091394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.091373 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.091455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.091423 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbql\" (UniqueName: \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.091605 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.091584 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.156593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.156563 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-27114d-serving-cert\"" Oct 30 21:52:31.161023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.161002 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:52:31.193062 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.193014 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.193234 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.193079 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbql\" (UniqueName: \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.193234 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.193176 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.193375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.193357 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.193969 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:31.193541 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-serving-cert: secret "maas-api-e2e-shared-a-w3-27114d-serving-cert" not found Oct 30 21:52:31.193969 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:31.193611 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls podName:83cb8d26-102d-47e6-aa08-0322310c0884 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:31.693588609 +0000 UTC m=+2332.387070824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls") pod "maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" (UID: "83cb8d26-102d-47e6-aa08-0322310c0884") : secret "maas-api-e2e-shared-a-w3-27114d-serving-cert" not found Oct 30 21:52:31.194636 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.194606 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.206249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.206223 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.253101 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.253074 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbql\" (UniqueName: \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.288367 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.288338 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:31.519992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.519959 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:31.699678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.699648 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.702229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.702209 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:31.965965 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:31.965934 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:32.346328 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:32.346300 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:52:32.347064 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:32.347031 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cb8d26_102d_47e6_aa08_0322310c0884.slice/crio-4ea1d0b74f9456628c0f3ac06f4321ea239dcd8be815f3e33d6c00e380bbde3f WatchSource:0}: Error finding container 4ea1d0b74f9456628c0f3ac06f4321ea239dcd8be815f3e33d6c00e380bbde3f: Status 404 returned error can't find the container with id 4ea1d0b74f9456628c0f3ac06f4321ea239dcd8be815f3e33d6c00e380bbde3f Oct 30 21:52:32.611982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:32.611948 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" event={"ID":"83cb8d26-102d-47e6-aa08-0322310c0884","Type":"ContainerStarted","Data":"4ea1d0b74f9456628c0f3ac06f4321ea239dcd8be815f3e33d6c00e380bbde3f"} Oct 30 21:52:33.623275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:33.623242 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" event={"ID":"83cb8d26-102d-47e6-aa08-0322310c0884","Type":"ContainerStarted","Data":"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6"} Oct 30 21:52:33.623666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:33.623425 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:33.737642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:33.737589 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" podStartSLOduration=2.864882437 podStartE2EDuration="3.737577355s" totalImagesPullingTime="872.694918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:33.735120487 +0000 UTC m=+2334.428602742" watchObservedRunningTime="2026-10-30 21:52:33.737577355 +0000 UTC m=+2334.431059591" Oct 30 21:52:35.522830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.522791 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86"] Oct 30 21:52:35.527485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.527463 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.627928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.627893 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86"] Oct 30 21:52:35.649753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.649681 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.649957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.649814 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.649957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.649910 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.649972 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46j4\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-kube-api-access-m46j4\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650073 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.650002 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.650136 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650253 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.650236 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.650369 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.650455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.650410 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.751567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751529 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.751820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751585 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.751820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751651 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.751820 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751687 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751829 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.751980 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46j4\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-kube-api-access-m46j4\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752041 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752238 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752238 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752117 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752238 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752203 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.752392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752131 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.753054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752822 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.753054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.752989 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.753423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.753394 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.754633 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.754607 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.755283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.755257 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.815354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.815292 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.815354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.815314 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46j4\" (UniqueName: \"kubernetes.io/projected/b85f2df0-2e73-44aa-9f4f-caf9bec10c71-kube-api-access-m46j4\") pod \"maas-default-gateway-openshift-default-67bf986f99-qzk86\" (UID: \"b85f2df0-2e73-44aa-9f4f-caf9bec10c71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:35.842336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:35.842313 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:36.234530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.234494 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:36.283904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.283868 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86"] Oct 30 21:52:36.310253 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:36.310221 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85f2df0_2e73_44aa_9f4f_caf9bec10c71.slice/crio-931454fadeed447f5a718719158a998f8ee6015e20425edfd01ec61e941fb13e WatchSource:0}: Error finding container 931454fadeed447f5a718719158a998f8ee6015e20425edfd01ec61e941fb13e: Status 404 returned error can't find the container with id 931454fadeed447f5a718719158a998f8ee6015e20425edfd01ec61e941fb13e Oct 30 21:52:36.313247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.313211 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:36.313469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.313301 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:36.313469 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.313376 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:36.653142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.653105 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" event={"ID":"b85f2df0-2e73-44aa-9f4f-caf9bec10c71","Type":"ContainerStarted","Data":"a6199f5680ae7a0544bdd1d2ac81c9c7185b49ac92bc2d70ea363546ca259fab"} Oct 30 21:52:36.653546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.653168 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" event={"ID":"b85f2df0-2e73-44aa-9f4f-caf9bec10c71","Type":"ContainerStarted","Data":"931454fadeed447f5a718719158a998f8ee6015e20425edfd01ec61e941fb13e"} Oct 30 21:52:36.777494 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.777442 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podStartSLOduration=1.7774238489999998 podStartE2EDuration="1.777423849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:36.771206376 +0000 UTC m=+2337.464688614" watchObservedRunningTime="2026-10-30 21:52:36.777423849 +0000 UTC m=+2337.470906085" Oct 30 21:52:36.842807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:36.842773 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:37.845417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:37.845381 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:37.845867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:37.845432 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:38.843622 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:38.843583 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:38.843827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:38.843646 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:39.636877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:39.636840 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:52:39.843621 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:39.843589 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:39.843808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:39.843650 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:40.843316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:40.843276 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:40.843761 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:40.843335 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:40.931135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:40.931101 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.842738 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:41.842686 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:41.842932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:41.842754 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:42.842838 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:42.842805 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:42.843226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:42.842858 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:42.889493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:42.889457 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:43.618318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:43.618282 4918 scope.go:122] "RemoveContainer" containerID="1d6d4478cf8db831fd4afe81f356b1b9bbe3105b69aac814c14f01ba829ef67d" Oct 30 21:52:43.842821 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:43.842783 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:52:43.843008 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:43.842841 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:52:45.530735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:45.527375 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:45.666745 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:45.666673 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:45.667170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:45.667153 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:45.668224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:45.668109 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" Oct 30 21:52:45.824314 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:45.824225 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:52:49.901488 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:49.901456 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:52:49.908618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:49.908597 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:49.999541 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:49.999430 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmvm\" (UniqueName: \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:49.999986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:49.999955 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.000310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.000268 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.000310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.000334 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.007108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.007083 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-737a64-serving-cert\"" Oct 30 21:52:50.014258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.014233 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:52:50.101377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.101344 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.101531 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.101387 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.101531 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.101437 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmvm\" (UniqueName: \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.101531 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.101463 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.101732 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:50.101571 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-serving-cert: secret "maas-api-e2e-api-a-737a64-serving-cert" not found Oct 30 21:52:50.101732 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:50.101646 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls podName:f31eafed-0196-42fd-82ff-8c56442fda11 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:50.601625896 +0000 UTC m=+2351.295108123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls") pod "maas-api-e2e-api-a-737a64-845d85c79c-rckbq" (UID: "f31eafed-0196-42fd-82ff-8c56442fda11") : secret "maas-api-e2e-api-a-737a64-serving-cert" not found Oct 30 21:52:50.102235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.102214 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.103828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.103809 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.129209 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.129188 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.160665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.160579 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmvm\" (UniqueName: \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.315198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.315155 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.509735 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.509614 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.618161 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.618127 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.620550 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.620528 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") pod \"maas-api-e2e-api-a-737a64-845d85c79c-rckbq\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:50.823170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:50.823094 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:51.199532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.199500 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:52:51.200292 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:52:51.200255 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31eafed_0196_42fd_82ff_8c56442fda11.slice/crio-43eac27baf4805ef19704a9f0977c5cec05a5ef35ac4d0270f1279d6f38f801d WatchSource:0}: Error finding container 43eac27baf4805ef19704a9f0977c5cec05a5ef35ac4d0270f1279d6f38f801d: Status 404 returned error can't find the container with id 43eac27baf4805ef19704a9f0977c5cec05a5ef35ac4d0270f1279d6f38f801d Oct 30 21:52:51.238224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.238198 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:52:51.238394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.238374 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" containerID="cri-o://c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee" gracePeriod=30 Oct 30 21:52:51.246520 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.246496 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:52:51.246731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.246672 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" containerID="cri-o://4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5" gracePeriod=30 Oct 30 21:52:51.283235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.283207 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:52:51.283441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.283417 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" containerID="cri-o://d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14" gracePeriod=30 Oct 30 21:52:51.287341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.287314 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:52:51.287538 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.287500 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" containerID="cri-o://fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327" gracePeriod=30 Oct 30 21:52:51.823070 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:51.823035 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" event={"ID":"f31eafed-0196-42fd-82ff-8c56442fda11","Type":"ContainerStarted","Data":"43eac27baf4805ef19704a9f0977c5cec05a5ef35ac4d0270f1279d6f38f801d"} Oct 30 21:52:52.516059 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:52.515968 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:52.516425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:52.516053 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:52.860831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:52.855091 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" event={"ID":"f31eafed-0196-42fd-82ff-8c56442fda11","Type":"ContainerStarted","Data":"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741"} Oct 30 21:52:52.860831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:52.855854 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:52.996816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:52.996768 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" podStartSLOduration=3.084677814 podStartE2EDuration="3.996752602s" totalImagesPullingTime="912.074788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:52.991983178 +0000 UTC m=+2353.685465415" watchObservedRunningTime="2026-10-30 21:52:52.996752602 +0000 UTC m=+2353.690234838" Oct 30 21:52:55.168942 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:55.168901 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:55.614341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:55.614305 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:55.614491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:55.614365 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:56.602984 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.602953 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:52:56.630606 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.630564 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:52:56.643313 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.643290 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:52:56.696240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.696218 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:52:56.698233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698211 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698239 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698331 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698368 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698485 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698462 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.698546 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698503 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698614 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698645 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:56.698674 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698647 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.698805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698686 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698728 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698761 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\") pod \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\" (UID: \"5db7213c-a8ca-48af-9a72-fe35e52e4ed8\") " Oct 30 21:52:56.698955 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.698909 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.699125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699099 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.699424 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699401 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.699527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699429 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.699527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699446 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.699527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699462 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.699527 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.699476 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.700854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.700823 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "kube-api-access-6f4tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.700854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.700834 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.701002 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.700904 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.701002 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.700944 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo" pod "5db7213c-a8ca-48af-9a72-fe35e52e4ed8" (UID: "5db7213c-a8ca-48af-9a72-fe35e52e4ed8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:56.800336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800255 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.800336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800333 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.800559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800356 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.800559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800540 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.800701 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800670 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.800780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800762 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.800836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800786 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:56.800899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800837 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.800899 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800869 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.800999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800900 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.800999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800934 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.800999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.800972 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.801154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801002 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.801154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801043 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.801154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801064 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:56.801154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801147 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801218 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801260 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801292 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801323 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801349 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801356 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801455 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801469 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801499 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801546 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801586 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801621 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\") pod \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\" (UID: \"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801674 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\") pod \"db9cec11-4398-49ad-b891-54d5450f1624\" (UID: \"db9cec11-4398-49ad-b891-54d5450f1624\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801685 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801755 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801829 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.801879 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\") pod \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\" (UID: \"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c\") " Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802196 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802422 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802519 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802728 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802768 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802789 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802813 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802830 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802847 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802863 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802883 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802898 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cec11-4398-49ad-b891-54d5450f1624-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802914 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802930 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4tk\" (UniqueName: \"kubernetes.io/projected/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-kube-api-access-6f4tk\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802946 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7213c-a8ca-48af-9a72-fe35e52e4ed8-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802963 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.802978 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803190 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803235 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803404 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803614 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803544 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803781 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.803839 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.804261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.804086 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "kube-api-access-8h9rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.807138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.806522 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "kube-api-access-pcxxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.807138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.806674 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.807254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.807200 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "kube-api-access-fp7x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.807254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.807202 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.807365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.807317 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:56.807365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.807343 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo" pod "db9cec11-4398-49ad-b891-54d5450f1624" (UID: "db9cec11-4398-49ad-b891-54d5450f1624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:56.808135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.808112 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:56.808369 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.808346 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy" pod "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" (UID: "ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.808676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.808656 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token" pod "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" (UID: "a15a6245-4fa5-4b77-ae08-d6e8e8cf725c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.885446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.885412 4918 generic.go:356] "Generic (PLEG): container finished" podID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerID="c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee" exitCode=0 Oct 30 21:52:56.885614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.885489 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" Oct 30 21:52:56.885614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.885498 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" event={"ID":"5db7213c-a8ca-48af-9a72-fe35e52e4ed8","Type":"ContainerDied","Data":"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee"} Oct 30 21:52:56.885614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.885580 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb" event={"ID":"5db7213c-a8ca-48af-9a72-fe35e52e4ed8","Type":"ContainerDied","Data":"cab77f6acc02bd68489f6a2f2e6fc61f1cff5796829faea5b4e0bcd0bd93aa1f"} Oct 30 21:52:56.885852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.885611 4918 scope.go:122] "RemoveContainer" containerID="c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee" Oct 30 21:52:56.893775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.893749 4918 generic.go:356] "Generic (PLEG): container finished" podID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerID="d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14" exitCode=0 Oct 30 21:52:56.893916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.893809 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" Oct 30 21:52:56.893916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.893828 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" event={"ID":"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9","Type":"ContainerDied","Data":"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14"} Oct 30 21:52:56.893916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.893881 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx" event={"ID":"ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9","Type":"ContainerDied","Data":"829d6933bd57ab2fc8c623f70b81e7f3638840e39e46bf742b40f1c29b0b6c78"} Oct 30 21:52:56.896586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.896555 4918 generic.go:356] "Generic (PLEG): container finished" podID="db9cec11-4398-49ad-b891-54d5450f1624" containerID="4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5" exitCode=0 Oct 30 21:52:56.896586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.896599 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" Oct 30 21:52:56.896869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.896637 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" event={"ID":"db9cec11-4398-49ad-b891-54d5450f1624","Type":"ContainerDied","Data":"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5"} Oct 30 21:52:56.896869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.896759 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl" event={"ID":"db9cec11-4398-49ad-b891-54d5450f1624","Type":"ContainerDied","Data":"be49689f9c19abf2d4489eea2253977577cbc505f61e0df58611013a5dc2f3e6"} Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.899444 4918 generic.go:356] "Generic (PLEG): container finished" podID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerID="fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327" exitCode=0 Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.899540 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.901347 4918 scope.go:122] "RemoveContainer" containerID="c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee" Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:56.902822 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee\": container with ID starting with c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee not found: ID does not exist" containerID="c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee" Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.902857 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee"} err="failed to get container status \"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee\": rpc error: code = NotFound desc = could not find container \"c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee\": container with ID starting with c0b44f68b2ac1802e709c8c053599c7dd511ba319dc39ff83cdc135b937009ee not found: ID does not exist" Oct 30 21:52:56.903753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.902879 4918 scope.go:122] "RemoveContainer" containerID="d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14" Oct 30 21:52:56.904245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904220 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904252 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904268 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904282 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904297 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h9rk\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-kube-api-access-8h9rk\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904311 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904325 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904339 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cec11-4398-49ad-b891-54d5450f1624-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904353 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904367 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcxxc\" (UniqueName: \"kubernetes.io/projected/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-kube-api-access-pcxxc\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904382 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7x8\" (UniqueName: \"kubernetes.io/projected/db9cec11-4398-49ad-b891-54d5450f1624-kube-api-access-fp7x8\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904397 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904411 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904425 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cec11-4398-49ad-b891-54d5450f1624-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904438 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904453 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904466 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.904515 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.904480 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.905768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.899475 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" event={"ID":"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c","Type":"ContainerDied","Data":"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327"} Oct 30 21:52:56.905935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.905915 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk" event={"ID":"a15a6245-4fa5-4b77-ae08-d6e8e8cf725c","Type":"ContainerDied","Data":"23ccb12420ef5384c594ca31416de30cb00c0e84b5a4f928ba792dc7d20374cc"} Oct 30 21:52:56.924467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.924446 4918 scope.go:122] "RemoveContainer" containerID="d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14" Oct 30 21:52:56.924834 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:56.924807 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14\": container with ID starting with d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14 not found: ID does not exist" containerID="d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14" Oct 30 21:52:56.924921 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.924845 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14"} err="failed to get container status \"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14\": rpc error: code = NotFound desc = could not find container \"d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14\": container with ID starting with d8d583d903c1cd787790e3e830d405145b9ec5f9c4872d7a98ee028e86af8a14 not found: ID does not exist" Oct 30 21:52:56.924921 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.924869 4918 scope.go:122] "RemoveContainer" containerID="4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5" Oct 30 21:52:56.937284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.937198 4918 scope.go:122] "RemoveContainer" containerID="4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5" Oct 30 21:52:56.937514 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:56.937486 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5\": container with ID starting with 4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5 not found: ID does not exist" containerID="4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5" Oct 30 21:52:56.937593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.937524 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5"} err="failed to get container status \"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5\": rpc error: code = NotFound desc = could not find container \"4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5\": container with ID starting with 4da6773fdc4f6635d7863f497891dc18c3f852fb81fa1c83f1b98640ceb2c9a5 not found: ID does not exist" Oct 30 21:52:56.937593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.937547 4918 scope.go:122] "RemoveContainer" containerID="fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327" Oct 30 21:52:56.948228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.948209 4918 scope.go:122] "RemoveContainer" containerID="fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327" Oct 30 21:52:56.948475 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:52:56.948457 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327\": container with ID starting with fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327 not found: ID does not exist" containerID="fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327" Oct 30 21:52:56.948524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:56.948481 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327"} err="failed to get container status \"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327\": rpc error: code = NotFound desc = could not find container \"fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327\": container with ID starting with fa26ec2295abeea329f9518b77ac776ebdd0d88829a95b08d0ff8076e64d6327 not found: ID does not exist" Oct 30 21:52:57.229339 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.225360 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:52:57.232530 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.232501 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-mxmpl"] Oct 30 21:52:57.333362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.333331 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:52:57.339564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.339542 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-z9ssk"] Oct 30 21:52:57.430455 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.430428 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" path="/var/lib/kubelet/pods/a15a6245-4fa5-4b77-ae08-d6e8e8cf725c/volumes" Oct 30 21:52:57.430992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.430969 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9cec11-4398-49ad-b891-54d5450f1624" path="/var/lib/kubelet/pods/db9cec11-4398-49ad-b891-54d5450f1624/volumes" Oct 30 21:52:57.443428 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.443401 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:52:57.447830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.447808 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-2gfjx"] Oct 30 21:52:57.564071 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.563987 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:52:57.576816 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:57.576792 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-x57wb"] Oct 30 21:52:58.868398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:58.868367 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:52:59.432031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.431996 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" path="/var/lib/kubelet/pods/5db7213c-a8ca-48af-9a72-fe35e52e4ed8/volumes" Oct 30 21:52:59.432584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.432562 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" path="/var/lib/kubelet/pods/ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9/volumes" Oct 30 21:52:59.549787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.549755 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:52:59.550470 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550453 4918 cpu_manager.go:414] "containerMap: removing container" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550472 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550503 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550508 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550521 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550527 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550543 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" Oct 30 21:52:59.550553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550548 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" Oct 30 21:52:59.550790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550644 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db7213c-a8ca-48af-9a72-fe35e52e4ed8" containerName="istio-proxy" Oct 30 21:52:59.550790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550653 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15a6245-4fa5-4b77-ae08-d6e8e8cf725c" containerName="istio-proxy" Oct 30 21:52:59.550790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550662 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae45481e-43e1-4ba5-8ac6-5bfade5ae1b9" containerName="istio-proxy" Oct 30 21:52:59.550790 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.550669 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9cec11-4398-49ad-b891-54d5450f1624" containerName="istio-proxy" Oct 30 21:52:59.555887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.555867 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.606225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.606202 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-2aa268-serving-cert\"" Oct 30 21:52:59.657354 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.657323 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:52:59.740043 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.739966 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.740203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.740087 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.740203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.740151 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk6n\" (UniqueName: \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.740203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.740188 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.775380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.775352 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:59.841310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.841276 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.841473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.841342 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk6n\" (UniqueName: \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.841473 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.841391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.841623 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.841556 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.842118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.842091 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.844116 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.844094 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.844255 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.844234 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.910278 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.910242 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk6n\" (UniqueName: \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\") pod \"maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:52:59.973007 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:52:59.972969 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:00.171840 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:00.171806 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:53:00.552672 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:00.552630 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:53:00.553679 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:00.553651 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42889823_0309_4e14_924a_d0a673eb4ac0.slice/crio-6ca7b745f6ef96c66387dd1664ca3f5e6a978d6a97d9e75cacc8f66bfc4f28fd WatchSource:0}: Error finding container 6ca7b745f6ef96c66387dd1664ca3f5e6a978d6a97d9e75cacc8f66bfc4f28fd: Status 404 returned error can't find the container with id 6ca7b745f6ef96c66387dd1664ca3f5e6a978d6a97d9e75cacc8f66bfc4f28fd Oct 30 21:53:00.945426 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:00.945392 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" event={"ID":"42889823-0309-4e14-924a-d0a673eb4ac0","Type":"ContainerStarted","Data":"6ca7b745f6ef96c66387dd1664ca3f5e6a978d6a97d9e75cacc8f66bfc4f28fd"} Oct 30 21:53:01.956282 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:01.956241 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" event={"ID":"42889823-0309-4e14-924a-d0a673eb4ac0","Type":"ContainerStarted","Data":"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551"} Oct 30 21:53:01.956782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:01.956388 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:53:02.075106 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:02.075009 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" podStartSLOduration=2.155453073 podStartE2EDuration="3.07498929s" totalImagesPullingTime="919.536217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:02.069425521 +0000 UTC m=+2362.762907769" watchObservedRunningTime="2026-10-30 21:53:02.07498929 +0000 UTC m=+2362.768471528" Oct 30 21:53:04.527754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:04.527645 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:04.914082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:04.914045 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.420608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.420572 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 21:53:05.427463 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.427429 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.487897 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.487861 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-71c0c7-openshift-default-dockercfg-k64h9\"" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509345 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509393 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509433 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9l9r\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509475 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509508 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509526 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509546 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509583 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.510450 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.509668 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.542248 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.542206 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 21:53:05.543119 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.543091 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.611311 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611251 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611329 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611377 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611458 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611584 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611633 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611701 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9l9r\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611768 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.611888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611820 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.612189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611928 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.612189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.611962 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.612189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.612046 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.612189 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.612126 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.612528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.612509 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.614258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.614227 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.614384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.614358 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.671401 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.671317 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.671612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.671580 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9l9r\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\") pod \"e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:05.749999 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:05.749945 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:06.206508 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.202910 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 21:53:06.206884 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:06.206849 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8a9e9d_36f9_40a8_a031_34907e07baaf.slice/crio-85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95 WatchSource:0}: Error finding container 85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95: Status 404 returned error can't find the container with id 85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95 Oct 30 21:53:06.210247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.210209 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:06.210364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.210297 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:06.210364 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.210343 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:06.331350 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.331318 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:06.995668 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.995631 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerStarted","Data":"8decb4ed657a512d3a058ab20701254257788d1e234e463d301f0268b40a1f03"} Oct 30 21:53:06.996079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:06.995687 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerStarted","Data":"85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95"} Oct 30 21:53:07.113850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:07.113796 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podStartSLOduration=2.113778922 podStartE2EDuration="2.113778922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:07.111903654 +0000 UTC m=+2367.805385890" watchObservedRunningTime="2026-10-30 21:53:07.113778922 +0000 UTC m=+2367.807261157" Oct 30 21:53:07.750934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:07.750899 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:07.756613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:07.756589 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:07.968261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:07.968231 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:53:08.004076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:08.003994 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:08.005312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:08.005287 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:53:09.405297 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:09.402579 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:10.050104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:10.050071 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:10.244593 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:10.244518 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.311553 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.311515 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.517979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.517945 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.705974 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.705943 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:53:13.710805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.710780 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.771611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.771584 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:53:13.814160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.814024 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-71c0c7-dockercfg-vsnxx\"" Oct 30 21:53:13.814343 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.814198 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71c0c7\"" Oct 30 21:53:13.842664 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.842626 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.843483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.843454 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfvn\" (UniqueName: \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.919196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.919156 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:53:13.970934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.952352 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.970934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.952403 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfvn\" (UniqueName: \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.970934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.953412 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:13.996625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.996593 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:53:13.997161 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:13.996993 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.029398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.029364 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfvn\" (UniqueName: \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\") pod \"payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:14.029578 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.029560 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:14.053877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.053409 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.053877 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.053647 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rfl\" (UniqueName: \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.155893 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.155660 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.156084 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.155981 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rfl\" (UniqueName: \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.157233 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.157204 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.214740 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.214677 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rfl\" (UniqueName: \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\") pod \"payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.320351 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.320275 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:14.418987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.418962 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:53:14.699050 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:14.699019 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bf3095_687a_48a7_87d1_f9ce40a2ab22.slice/crio-a50c2130caf249f29f0547c2a1cc8092092bb929b348952cd2caebfc7c3872c5 WatchSource:0}: Error finding container a50c2130caf249f29f0547c2a1cc8092092bb929b348952cd2caebfc7c3872c5: Status 404 returned error can't find the container with id a50c2130caf249f29f0547c2a1cc8092092bb929b348952cd2caebfc7c3872c5 Oct 30 21:53:14.700333 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.700298 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:53:14.753676 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:14.752549 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:15.063967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.063881 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" event={"ID":"18a26b30-d49d-48b0-a35b-cbf402765197","Type":"ContainerStarted","Data":"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9"} Oct 30 21:53:15.063967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.063946 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" event={"ID":"18a26b30-d49d-48b0-a35b-cbf402765197","Type":"ContainerStarted","Data":"820a0c2e59f4d967d8a4ad082ce81ee80b5c81defd470525377fe0b3967d2045"} Oct 30 21:53:15.064182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.063976 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:15.065660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.065630 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" event={"ID":"d9bf3095-687a-48a7-87d1-f9ce40a2ab22","Type":"ContainerStarted","Data":"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465"} Oct 30 21:53:15.065787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.065682 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" event={"ID":"d9bf3095-687a-48a7-87d1-f9ce40a2ab22","Type":"ContainerStarted","Data":"a50c2130caf249f29f0547c2a1cc8092092bb929b348952cd2caebfc7c3872c5"} Oct 30 21:53:15.065787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.065724 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:15.181508 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.181447 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" podStartSLOduration=2.181422273 podStartE2EDuration="2.181422273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:15.178256618 +0000 UTC m=+2375.871738854" watchObservedRunningTime="2026-10-30 21:53:15.181422273 +0000 UTC m=+2375.874904510" Oct 30 21:53:15.325197 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:15.325095 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" podStartSLOduration=2.325076766 podStartE2EDuration="2.325076766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:15.296451181 +0000 UTC m=+2375.989933417" watchObservedRunningTime="2026-10-30 21:53:15.325076766 +0000 UTC m=+2376.018559002" Oct 30 21:53:15.804249 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:53:15.804208 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:53:17.051031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:17.050997 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:17.551261 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:17.551224 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:18.210404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:18.209138 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:18.697447 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:18.697405 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:20.185436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:20.185341 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.226812 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:23.226767 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.552957 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:23.552851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.781665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:23.781627 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.789559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:24.789526 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.972126 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:24.972092 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:25.323232 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:25.323196 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:26.078578 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:26.078545 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:53:26.079121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:26.078859 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:53:26.628768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:26.628701 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:26.832661 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:26.832630 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:27.004275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.004199 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 21:53:27.011690 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.011665 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.061059 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.061036 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ca0a55-dockercfg-xl9rm\"" Oct 30 21:53:27.061059 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.061040 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ca0a55\"" Oct 30 21:53:27.061518 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.061497 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 21:53:27.097825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.096231 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgrc\" (UniqueName: \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.097825 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.096330 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.166108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.166073 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 21:53:27.182028 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.181996 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.197584 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.197548 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgrc\" (UniqueName: \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.197779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.197653 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.198408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.198386 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.216828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.216799 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 21:53:27.256400 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.256328 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgrc\" (UniqueName: \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\") pod \"payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.300559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.300522 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nd66\" (UniqueName: \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.300768 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.300672 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.329256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.329229 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:27.403179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.402649 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.403179 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.402932 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nd66\" (UniqueName: \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.405837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.403543 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.462467 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.462439 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nd66\" (UniqueName: \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\") pod \"payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.502613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.502582 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:27.722613 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.722586 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 21:53:27.723290 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:27.723262 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faeebe6_b203_4144_955c_914bf5a968e4.slice/crio-59deda55ce8ab8816da802e5443c5e032c219ff8ce72c4bab7d4d2bd14e265ac WatchSource:0}: Error finding container 59deda55ce8ab8816da802e5443c5e032c219ff8ce72c4bab7d4d2bd14e265ac: Status 404 returned error can't find the container with id 59deda55ce8ab8816da802e5443c5e032c219ff8ce72c4bab7d4d2bd14e265ac Oct 30 21:53:27.884700 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:27.884669 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 21:53:27.886801 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:27.886777 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10adf94a_3b9a_45bc_aca9_bbb9f45b0f1e.slice/crio-eb563270f0635e3393806c48f0a0c88669f9f639311d686185f25dcacaae0e57 WatchSource:0}: Error finding container eb563270f0635e3393806c48f0a0c88669f9f639311d686185f25dcacaae0e57: Status 404 returned error can't find the container with id eb563270f0635e3393806c48f0a0c88669f9f639311d686185f25dcacaae0e57 Oct 30 21:53:28.191780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.191735 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" event={"ID":"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e","Type":"ContainerStarted","Data":"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c"} Oct 30 21:53:28.192480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.191804 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" event={"ID":"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e","Type":"ContainerStarted","Data":"eb563270f0635e3393806c48f0a0c88669f9f639311d686185f25dcacaae0e57"} Oct 30 21:53:28.192480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.191864 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:28.194125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.194015 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" event={"ID":"4faeebe6-b203-4144-955c-914bf5a968e4","Type":"ContainerStarted","Data":"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb"} Oct 30 21:53:28.194125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.194059 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" event={"ID":"4faeebe6-b203-4144-955c-914bf5a968e4","Type":"ContainerStarted","Data":"59deda55ce8ab8816da802e5443c5e032c219ff8ce72c4bab7d4d2bd14e265ac"} Oct 30 21:53:28.194125 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.194094 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:28.307170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.307115 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" podStartSLOduration=1.307102021 podStartE2EDuration="1.307102021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:28.302148833 +0000 UTC m=+2388.995631069" watchObservedRunningTime="2026-10-30 21:53:28.307102021 +0000 UTC m=+2389.000584256" Oct 30 21:53:28.422358 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:28.422285 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" podStartSLOduration=2.422264807 podStartE2EDuration="2.422264807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:28.411203464 +0000 UTC m=+2389.104685701" watchObservedRunningTime="2026-10-30 21:53:28.422264807 +0000 UTC m=+2389.115747043" Oct 30 21:53:30.306137 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:30.305161 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:35.377770 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:35.377540 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:39.204970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.204939 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 21:53:39.205503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.205025 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 21:53:39.800540 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.800509 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:53:39.814168 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.814142 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:53:39.816917 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.816890 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:53:39.822719 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.822683 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:53:39.828976 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.828956 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:53:39.834392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:39.834374 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:53:40.333625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:40.333585 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:40.947009 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:40.946963 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:41.402061 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:41.402034 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:42.090621 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:42.090579 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:42.415905 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:42.415867 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:43.205031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:43.204996 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:43.436848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:43.436813 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:43.635580 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:43.635545 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:45.515258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:45.515215 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:45.805163 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:53:45.805075 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:53:46.575129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:46.575097 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:46.925694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:46.925644 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:47.232291 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:47.232208 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.856528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:49.856493 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.084026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.083988 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.369797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.369766 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:53:50.377359 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.377331 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.435692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.435652 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:53:50.484505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.484466 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ed6135\"" Oct 30 21:53:50.489500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.485066 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ed6135-dockercfg-56qtk\"" Oct 30 21:53:50.537265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.537234 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvlw\" (UniqueName: \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.537458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.537392 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.552394 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.552351 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:53:50.557092 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.557061 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.615202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.615166 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:53:50.638805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.638728 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvlw\" (UniqueName: \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.638805 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.638800 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.639031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.638976 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.639096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.639035 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzh9\" (UniqueName: \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.639776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.639752 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.697310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.697276 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvlw\" (UniqueName: \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\") pod \"payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.708123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.708094 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:50.740193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.740162 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzh9\" (UniqueName: \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.740325 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.740285 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.741193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.741167 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.812731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.812634 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzh9\" (UniqueName: \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\") pod \"payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:50.872843 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:50.872806 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:51.059500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.059460 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.093295 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.093262 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:53:51.098453 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:51.098419 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd247f88e_0b87_49be_a775_3454fbf362f6.slice/crio-04a86b4c5e4384d04901f1f7195101636700ccc3aea938ba35968425996e9e1e WatchSource:0}: Error finding container 04a86b4c5e4384d04901f1f7195101636700ccc3aea938ba35968425996e9e1e: Status 404 returned error can't find the container with id 04a86b4c5e4384d04901f1f7195101636700ccc3aea938ba35968425996e9e1e Oct 30 21:53:51.253958 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.253923 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:53:51.254107 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:53:51.254081 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981fcd7c_0a93_4acb_aa01_2d0cf6eb4576.slice/crio-37ff4ae74d4d820e39cfb70c72d92411ffa079dad0991ded037a4d679039dd71 WatchSource:0}: Error finding container 37ff4ae74d4d820e39cfb70c72d92411ffa079dad0991ded037a4d679039dd71: Status 404 returned error can't find the container with id 37ff4ae74d4d820e39cfb70c72d92411ffa079dad0991ded037a4d679039dd71 Oct 30 21:53:51.385332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.385198 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" event={"ID":"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576","Type":"ContainerStarted","Data":"e8abb7810dec676bf8f12025b37521892f2d337adfa1b10afd4b0f982c69bc5c"} Oct 30 21:53:51.385332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.385267 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" event={"ID":"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576","Type":"ContainerStarted","Data":"37ff4ae74d4d820e39cfb70c72d92411ffa079dad0991ded037a4d679039dd71"} Oct 30 21:53:51.385332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.385300 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:53:51.387303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.387271 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" event={"ID":"d247f88e-0b87-49be-a775-3454fbf362f6","Type":"ContainerStarted","Data":"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c"} Oct 30 21:53:51.387441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.387329 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" event={"ID":"d247f88e-0b87-49be-a775-3454fbf362f6","Type":"ContainerStarted","Data":"04a86b4c5e4384d04901f1f7195101636700ccc3aea938ba35968425996e9e1e"} Oct 30 21:53:51.387441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.387359 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:53:51.504404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.504305 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" podStartSLOduration=1.504287777 podStartE2EDuration="1.504287777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.498586028 +0000 UTC m=+2412.192068264" watchObservedRunningTime="2026-10-30 21:53:51.504287777 +0000 UTC m=+2412.197770014" Oct 30 21:53:51.622760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:51.622027 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" podStartSLOduration=1.622007806 podStartE2EDuration="1.622007806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.619119567 +0000 UTC m=+2412.312601837" watchObservedRunningTime="2026-10-30 21:53:51.622007806 +0000 UTC m=+2412.315490043" Oct 30 21:53:52.187275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:52.187221 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:53.675798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:53.672207 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:53.877208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:53.877163 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:56.150545 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:56.150507 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:59.024878 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:59.024845 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:59.282637 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:59.282554 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:59.517251 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:53:59.517213 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:01.772533 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:01.772496 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.388618 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.388582 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.398056 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.398032 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:54:02.398206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.398105 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:54:02.593507 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.593469 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.795200 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.795117 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:54:02.802989 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.802927 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:02.853579 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.853552 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7d7f57\"" Oct 30 21:54:02.854833 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.854808 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:54:02.884255 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.884214 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7d7f57-dockercfg-bwd9s\"" Oct 30 21:54:02.911760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.911723 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4zg\" (UniqueName: \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:02.912304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.912254 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:02.972220 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.972186 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:54:02.978045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.978020 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:02.998504 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:02.998471 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:54:03.013985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.013954 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4zg\" (UniqueName: \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:03.014123 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.014093 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:03.015412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.015382 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:03.072187 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.072115 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4zg\" (UniqueName: \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\") pod \"payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:03.115172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.115139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.115340 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.115182 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8l5\" (UniqueName: \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.121504 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.121479 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:03.219985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.217585 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.219985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.217627 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8l5\" (UniqueName: \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.219985 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.218611 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.276425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.276344 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8l5\" (UniqueName: \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\") pod \"payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.294158 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.294119 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:03.503297 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:03.503260 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2220fea4_1ae3_4a51_b028_e912df0d4001.slice/crio-3387acecfce00e2601a5df36fae776efca631806aecb0692170082ed7672b2e2 WatchSource:0}: Error finding container 3387acecfce00e2601a5df36fae776efca631806aecb0692170082ed7672b2e2: Status 404 returned error can't find the container with id 3387acecfce00e2601a5df36fae776efca631806aecb0692170082ed7672b2e2 Oct 30 21:54:03.505620 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.505587 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:54:03.514827 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.514803 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" event={"ID":"2220fea4-1ae3-4a51-b028-e912df0d4001","Type":"ContainerStarted","Data":"3387acecfce00e2601a5df36fae776efca631806aecb0692170082ed7672b2e2"} Oct 30 21:54:03.678050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:03.678024 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:54:03.678251 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:03.678231 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7393e77_09f5_406c_a781_4af580134e07.slice/crio-98c49f1c6d648ede5ea607ab09bc7ec64e0955b9ac4a77247192f6e3cc1ae157 WatchSource:0}: Error finding container 98c49f1c6d648ede5ea607ab09bc7ec64e0955b9ac4a77247192f6e3cc1ae157: Status 404 returned error can't find the container with id 98c49f1c6d648ede5ea607ab09bc7ec64e0955b9ac4a77247192f6e3cc1ae157 Oct 30 21:54:04.525486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.525447 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" event={"ID":"2220fea4-1ae3-4a51-b028-e912df0d4001","Type":"ContainerStarted","Data":"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5"} Oct 30 21:54:04.525953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.525517 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:04.527380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.527350 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" event={"ID":"c7393e77-09f5-406c-a781-4af580134e07","Type":"ContainerStarted","Data":"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469"} Oct 30 21:54:04.527524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.527400 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" event={"ID":"c7393e77-09f5-406c-a781-4af580134e07","Type":"ContainerStarted","Data":"98c49f1c6d648ede5ea607ab09bc7ec64e0955b9ac4a77247192f6e3cc1ae157"} Oct 30 21:54:04.527524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.527426 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:04.649094 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.649036 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" podStartSLOduration=2.64901755 podStartE2EDuration="2.64901755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:04.642750327 +0000 UTC m=+2425.336232564" watchObservedRunningTime="2026-10-30 21:54:04.64901755 +0000 UTC m=+2425.342499789" Oct 30 21:54:04.772356 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:04.772298 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" podStartSLOduration=2.772284088 podStartE2EDuration="2.772284088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:04.757867823 +0000 UTC m=+2425.451350075" watchObservedRunningTime="2026-10-30 21:54:04.772284088 +0000 UTC m=+2425.465766323" Oct 30 21:54:05.346087 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:05.346056 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.121144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:06.121115 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.947042 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:06.947001 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:07.166434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.166401 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 21:54:07.177131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.177103 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.239491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.239432 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2df7b6-openshift-default-dockercfg-bbhvx\"" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294276 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294329 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294395 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294454 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294501 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294551 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294633 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294640 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2lk\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294781 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.299726 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.294846 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.324769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.323191 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:07.396464 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396419 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396495 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396540 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396579 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396657 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396621 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396693 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2lk\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396738 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.396900 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396789 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.397049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396910 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.397049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.396917 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.397300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.397276 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.397498 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.397480 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.398098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.398073 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.398265 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.398116 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.405751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.405237 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.406047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.406024 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.459015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.458985 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.461417 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.461387 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2lk\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\") pod \"e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.496408 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.496315 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:07.523575 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.523535 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:07.947173 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:07.947143 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a553b9_fd1d_496b_9134_28b19ca0d850.slice/crio-c7cd0e0d798c4cb1c4d869d9ec50a9bfa857ced2ba1d546ffba3dc2017e51ac2 WatchSource:0}: Error finding container c7cd0e0d798c4cb1c4d869d9ec50a9bfa857ced2ba1d546ffba3dc2017e51ac2: Status 404 returned error can't find the container with id c7cd0e0d798c4cb1c4d869d9ec50a9bfa857ced2ba1d546ffba3dc2017e51ac2 Oct 30 21:54:07.948274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.948249 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 21:54:07.949983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.949953 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:07.950069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.950018 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:07.950069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:07.950046 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:08.566412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:08.566374 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerStarted","Data":"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84"} Oct 30 21:54:08.566920 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:08.566424 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerStarted","Data":"c7cd0e0d798c4cb1c4d869d9ec50a9bfa857ced2ba1d546ffba3dc2017e51ac2"} Oct 30 21:54:08.697886 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:08.697817 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podStartSLOduration=1.6978011400000002 podStartE2EDuration="1.69780114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:08.696569634 +0000 UTC m=+2429.390051870" watchObservedRunningTime="2026-10-30 21:54:08.69780114 +0000 UTC m=+2429.391283376" Oct 30 21:54:09.497483 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:09.497445 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:09.504004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:09.503975 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:09.576015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:09.575986 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:09.576952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:09.576923 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:54:12.121425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:12.121381 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:12.533259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:12.533179 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:12.771523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:12.771484 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:15.537823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:15.537787 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:54:15.538350 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:15.537881 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:54:15.818624 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:15.818506 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:54:16.722336 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:16.722303 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.183349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.183313 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 21:54:17.189221 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.189200 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.227320 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.227297 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.227453 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.227424 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.227501 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.227484 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.227727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.227696 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm8d\" (UniqueName: \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.240564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.240543 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71c0c7-serving-cert\"" Oct 30 21:54:17.291749 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.291714 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 21:54:17.295863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.295832 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.329415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.329391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.329547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.329480 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.329547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.329498 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.329643 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.329620 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm8d\" (UniqueName: \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.329643 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:17.329629 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-serving-cert: secret "maas-api-e2e-api-a-71c0c7-serving-cert" not found Oct 30 21:54:17.329759 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:17.329746 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls podName:174885ad-eb95-4eea-87be-75479e6f118b nodeName:}" failed. No retries permitted until 2026-10-30 21:54:17.829687258 +0000 UTC m=+2438.523169491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls") pod "maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" (UID: "174885ad-eb95-4eea-87be-75479e6f118b") : secret "maas-api-e2e-api-a-71c0c7-serving-cert" not found Oct 30 21:54:17.330172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.330153 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.332093 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.332075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.400004 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.399964 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm8d\" (UniqueName: \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.494175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.494089 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.720407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.720373 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.842066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.841970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.845134 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.845101 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") pod \"maas-api-e2e-api-a-71c0c7-d846bd488-hndbq\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:17.949753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:17.949696 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:18.104896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.104819 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:18.168183 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.168158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:18.417722 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.417267 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:18.439057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.439027 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 21:54:18.445692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.445669 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.509991 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.509868 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 21:54:18.546052 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.546029 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2df7b6\"" Oct 30 21:54:18.546209 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.546085 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2df7b6-dockercfg-46vh5\"" Oct 30 21:54:18.552406 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.552377 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.552541 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.552525 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxvs\" (UniqueName: \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.553435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.553410 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 21:54:18.646145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.646108 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 21:54:18.652945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.652912 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.662526 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.662496 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" event={"ID":"174885ad-eb95-4eea-87be-75479e6f118b","Type":"ContainerStarted","Data":"5dbeb84a31e94fc39b681137e5ab91c142de79ab3c494efeaaf6d913a0b60b28"} Oct 30 21:54:18.663136 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.663106 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.663301 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.663211 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxvs\" (UniqueName: \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.664357 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.664332 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.705656 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.705577 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 21:54:18.722809 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.722777 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxvs\" (UniqueName: \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\") pod \"payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.761512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.761482 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:18.764216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.764187 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9r4\" (UniqueName: \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.764361 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.764305 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.865753 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.865679 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.866256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.865926 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9r4\" (UniqueName: \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.866673 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.866613 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.934624 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.934592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9r4\" (UniqueName: \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\") pod \"payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:18.976505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:18.976436 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:19.150628 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.150599 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 21:54:19.151525 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:19.151496 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacc135d_2181_4a53_a5b9_791939b29f72.slice/crio-4856df323ad66f3f14370697e660543f3f733b39309359c8e704c7f0412f7d0a WatchSource:0}: Error finding container 4856df323ad66f3f14370697e660543f3f733b39309359c8e704c7f0412f7d0a: Status 404 returned error can't find the container with id 4856df323ad66f3f14370697e660543f3f733b39309359c8e704c7f0412f7d0a Oct 30 21:54:19.377972 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.377942 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 21:54:19.378109 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:19.377986 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1a71dc_d561_46bc_9ddb_c2607217d015.slice/crio-56b7fa68e00e8666d30a6c8e84c6a0fe90c25b5604294508708446442e7be8b8 WatchSource:0}: Error finding container 56b7fa68e00e8666d30a6c8e84c6a0fe90c25b5604294508708446442e7be8b8: Status 404 returned error can't find the container with id 56b7fa68e00e8666d30a6c8e84c6a0fe90c25b5604294508708446442e7be8b8 Oct 30 21:54:19.673097 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.673012 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" event={"ID":"174885ad-eb95-4eea-87be-75479e6f118b","Type":"ContainerStarted","Data":"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40"} Oct 30 21:54:19.673254 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.673173 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:19.675181 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.675154 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" event={"ID":"0d1a71dc-d561-46bc-9ddb-c2607217d015","Type":"ContainerStarted","Data":"19bc4d5fb9d94b5c5e002b5a1559205aff66441b0ba4bc3a13a170673f6f9768"} Oct 30 21:54:19.675330 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.675192 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" event={"ID":"0d1a71dc-d561-46bc-9ddb-c2607217d015","Type":"ContainerStarted","Data":"56b7fa68e00e8666d30a6c8e84c6a0fe90c25b5604294508708446442e7be8b8"} Oct 30 21:54:19.675330 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.675230 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:19.677225 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.677201 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" event={"ID":"bacc135d-2181-4a53-a5b9-791939b29f72","Type":"ContainerStarted","Data":"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a"} Oct 30 21:54:19.677344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.677253 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:19.677344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.677279 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" event={"ID":"bacc135d-2181-4a53-a5b9-791939b29f72","Type":"ContainerStarted","Data":"4856df323ad66f3f14370697e660543f3f733b39309359c8e704c7f0412f7d0a"} Oct 30 21:54:19.797308 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.797262 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" podStartSLOduration=1.817867515 podStartE2EDuration="2.79724745s" totalImagesPullingTime="979.379935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:19.794608742 +0000 UTC m=+2440.488090978" watchObservedRunningTime="2026-10-30 21:54:19.79724745 +0000 UTC m=+2440.490729683" Oct 30 21:54:19.922138 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:19.922080 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" podStartSLOduration=1.922063057 podStartE2EDuration="1.922063057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:19.921974007 +0000 UTC m=+2440.615456245" watchObservedRunningTime="2026-10-30 21:54:19.922063057 +0000 UTC m=+2440.615545294" Oct 30 21:54:20.037670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:20.037552 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" podStartSLOduration=2.037532189 podStartE2EDuration="2.037532189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:20.03491553 +0000 UTC m=+2440.728397766" watchObservedRunningTime="2026-10-30 21:54:20.037532189 +0000 UTC m=+2440.731014426" Oct 30 21:54:22.098491 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:22.098456 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:22.359517 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:22.359482 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:23.105861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:23.105821 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:25.693503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:25.693470 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 21:54:28.362601 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.362563 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.587969 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.587937 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.804748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.804616 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.820776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.820741 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:54:28.826312 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.826291 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:28.877489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.877455 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-167561-dockercfg-dbpbb\"" Oct 30 21:54:28.932728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.926827 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ht2b\" (UniqueName: \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:28.932728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.926923 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:28.932728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:28.927330 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-167561\"" Oct 30 21:54:29.006728 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.006015 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:54:29.031024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.020953 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:54:29.031024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.028892 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ht2b\" (UniqueName: \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.031024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.028969 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.031024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.029971 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.055979 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.052997 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.130760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.130312 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9ct\" (UniqueName: \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.130760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.130461 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.132547 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.132516 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ht2b\" (UniqueName: \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\") pod \"payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.141024 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.140938 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.150030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.149055 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:54:29.231650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.231607 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.231844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.231822 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9ct\" (UniqueName: \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.233175 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.233150 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.294890 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.294820 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9ct\" (UniqueName: \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\") pod \"payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.509419 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.509383 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:29.549741 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:29.549683 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00e074a_962f_439a_afc6_8ba3c3abf1d3.slice/crio-eb12ca48b101c7ebd8f6031134eef3d9dd85e0dc852b76652ad7ce964674496b WatchSource:0}: Error finding container eb12ca48b101c7ebd8f6031134eef3d9dd85e0dc852b76652ad7ce964674496b: Status 404 returned error can't find the container with id eb12ca48b101c7ebd8f6031134eef3d9dd85e0dc852b76652ad7ce964674496b Oct 30 21:54:29.551268 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.551244 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:54:29.766375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.766280 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" event={"ID":"e00e074a-962f-439a-afc6-8ba3c3abf1d3","Type":"ContainerStarted","Data":"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567"} Oct 30 21:54:29.766375 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.766356 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:29.766566 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.766383 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" event={"ID":"e00e074a-962f-439a-afc6-8ba3c3abf1d3","Type":"ContainerStarted","Data":"eb12ca48b101c7ebd8f6031134eef3d9dd85e0dc852b76652ad7ce964674496b"} Oct 30 21:54:29.857031 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.857000 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:29.878155 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.878108 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" podStartSLOduration=1.87809169 podStartE2EDuration="1.87809169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:29.876256167 +0000 UTC m=+2450.569738403" watchObservedRunningTime="2026-10-30 21:54:29.87809169 +0000 UTC m=+2450.571573929" Oct 30 21:54:29.890451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:29.890424 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:54:29.891162 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:29.891134 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5b3991_546f_4cdd_ad4b_1d80657ab6c5.slice/crio-0298b5243950f9138c9b42a8ffdfd5999c67690c66a7afc97db5e28b3983846b WatchSource:0}: Error finding container 0298b5243950f9138c9b42a8ffdfd5999c67690c66a7afc97db5e28b3983846b: Status 404 returned error can't find the container with id 0298b5243950f9138c9b42a8ffdfd5999c67690c66a7afc97db5e28b3983846b Oct 30 21:54:30.690002 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:30.689974 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 21:54:30.690426 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:30.690053 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 21:54:30.787489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:30.787442 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" event={"ID":"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5","Type":"ContainerStarted","Data":"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856"} Oct 30 21:54:30.787669 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:30.787538 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" event={"ID":"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5","Type":"ContainerStarted","Data":"0298b5243950f9138c9b42a8ffdfd5999c67690c66a7afc97db5e28b3983846b"} Oct 30 21:54:31.026133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:31.026024 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" podStartSLOduration=3.026009612 podStartE2EDuration="3.026009612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:31.024728267 +0000 UTC m=+2451.718210514" watchObservedRunningTime="2026-10-30 21:54:31.026009612 +0000 UTC m=+2451.719491848" Oct 30 21:54:31.796721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:31.796681 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:32.944329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:32.944288 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:37.366914 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.366885 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 21:54:37.378050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.378016 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.433121 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.433091 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 21:54:37.441500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.441469 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.441642 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.441529 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.441777 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.441763 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr29z\" (UniqueName: \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.441866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.441830 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.468595 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.468567 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-ca0a55-serving-cert\"" Oct 30 21:54:37.542693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.542659 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr29z\" (UniqueName: \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.542870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.542783 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.542870 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.542849 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.542987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.542891 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.542987 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:37.542948 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-serving-cert: secret "maas-api-e2e-shared-b-w5-ca0a55-serving-cert" not found Oct 30 21:54:37.543076 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:37.543021 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls podName:f7591d4f-54e9-4a96-b72a-8a4ec57af991 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:38.042996969 +0000 UTC m=+2458.736479200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls") pod "maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" (UID: "f7591d4f-54e9-4a96-b72a-8a4ec57af991") : secret "maas-api-e2e-shared-b-w5-ca0a55-serving-cert" not found Oct 30 21:54:37.543524 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.543504 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.545403 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.545379 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.607625 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.607592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr29z\" (UniqueName: \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:37.618608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.618542 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:37.816003 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:37.815969 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:38.052399 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:38.052317 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:38.054819 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:38.054796 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:38.294298 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:38.294260 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:38.685562 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:38.685530 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 21:54:38.685978 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:38.685802 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7591d4f_54e9_4a96_b72a_8a4ec57af991.slice/crio-057ee076c1c3360a722f31d38546fefed28126e259dc47d37606ec5efe56b58c WatchSource:0}: Error finding container 057ee076c1c3360a722f31d38546fefed28126e259dc47d37606ec5efe56b58c: Status 404 returned error can't find the container with id 057ee076c1c3360a722f31d38546fefed28126e259dc47d37606ec5efe56b58c Oct 30 21:54:38.857987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:38.857955 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" event={"ID":"f7591d4f-54e9-4a96-b72a-8a4ec57af991","Type":"ContainerStarted","Data":"057ee076c1c3360a722f31d38546fefed28126e259dc47d37606ec5efe56b58c"} Oct 30 21:54:39.868500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:39.868400 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" event={"ID":"f7591d4f-54e9-4a96-b72a-8a4ec57af991","Type":"ContainerStarted","Data":"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29"} Oct 30 21:54:39.868500 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:39.868470 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:39.988534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:39.988485 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" podStartSLOduration=2.075877387 podStartE2EDuration="2.988471878s" totalImagesPullingTime="912.594491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:39.985618538 +0000 UTC m=+2460.679100775" watchObservedRunningTime="2026-10-30 21:54:39.988471878 +0000 UTC m=+2460.681954114" Oct 30 21:54:40.790894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:40.790867 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:54:41.114535 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.114504 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11379-3d498e"] Oct 30 21:54:41.119349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.119331 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:41.172219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.172200 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sxztf\"" Oct 30 21:54:41.181911 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.181884 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11379-3d498e"] Oct 30 21:54:41.196332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.196303 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2cgx\" (UniqueName: \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\") pod \"test-curl-11379-3d498e\" (UID: \"c1b517c8-106f-4fe3-b291-cbc3745fc5c0\") " pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:41.297692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.297655 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2cgx\" (UniqueName: \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\") pod \"test-curl-11379-3d498e\" (UID: \"c1b517c8-106f-4fe3-b291-cbc3745fc5c0\") " pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:41.356554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.356527 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2cgx\" (UniqueName: \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\") pod \"test-curl-11379-3d498e\" (UID: \"c1b517c8-106f-4fe3-b291-cbc3745fc5c0\") " pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:41.432872 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.432817 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:41.815561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.815525 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11379-3d498e"] Oct 30 21:54:41.816527 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:54:41.816503 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1b517c8_106f_4fe3_b291_cbc3745fc5c0.slice/crio-ffe86bc7b70f6f4a3ca25c074a7790f992af1d28109ba4b56f6b235400285ece WatchSource:0}: Error finding container ffe86bc7b70f6f4a3ca25c074a7790f992af1d28109ba4b56f6b235400285ece: Status 404 returned error can't find the container with id ffe86bc7b70f6f4a3ca25c074a7790f992af1d28109ba4b56f6b235400285ece Oct 30 21:54:41.892918 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:41.892875 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11379-3d498e" event={"ID":"c1b517c8-106f-4fe3-b291-cbc3745fc5c0","Type":"ContainerStarted","Data":"ffe86bc7b70f6f4a3ca25c074a7790f992af1d28109ba4b56f6b235400285ece"} Oct 30 21:54:42.743528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:42.743488 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:42.807679 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:42.807648 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:54:42.902743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:42.902678 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11379-3d498e" event={"ID":"c1b517c8-106f-4fe3-b291-cbc3745fc5c0","Type":"ContainerStarted","Data":"31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c"} Oct 30 21:54:43.039663 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:43.039567 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11379-3d498e" podStartSLOduration=2.039552804 podStartE2EDuration="2.039552804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:43.038501367 +0000 UTC m=+2463.731983603" watchObservedRunningTime="2026-10-30 21:54:43.039552804 +0000 UTC m=+2463.733035039" Oct 30 21:54:43.572667 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:43.572631 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:44.614666 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:44.614627 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11379-3d498e"] Oct 30 21:54:44.621068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:44.621040 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11379-3d498e"] Oct 30 21:54:44.920288 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:44.920203 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11379-3d498e" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" containerName="test-curl-11379-3d498e" containerID="cri-o://31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c" gracePeriod=2 Oct 30 21:54:44.970831 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:44.970792 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11379-3d498e\" is forbidden: User \"system:node:ip-10-0-132-23.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-23.us-west-2.compute.internal' and this object" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:45.806038 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:45.806001 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:54:45.882927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:45.882899 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 21:54:46.461398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:46.461362 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:47.185245 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.185182 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:47.278961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.278932 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\" (UniqueName: \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\") pod \"c1b517c8-106f-4fe3-b291-cbc3745fc5c0\" (UID: \"c1b517c8-106f-4fe3-b291-cbc3745fc5c0\") " Oct 30 21:54:47.281015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.280985 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx" pod "c1b517c8-106f-4fe3-b291-cbc3745fc5c0" (UID: "c1b517c8-106f-4fe3-b291-cbc3745fc5c0"). InnerVolumeSpecName "kube-api-access-q2cgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:47.381133 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.381107 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2cgx\" (UniqueName: \"kubernetes.io/projected/c1b517c8-106f-4fe3-b291-cbc3745fc5c0-kube-api-access-q2cgx\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:47.430171 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.430143 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" path="/var/lib/kubelet/pods/c1b517c8-106f-4fe3-b291-cbc3745fc5c0/volumes" Oct 30 21:54:47.641795 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.641762 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:47.949195 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.949162 4918 generic.go:356] "Generic (PLEG): container finished" podID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" containerID="31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c" exitCode=137 Oct 30 21:54:47.949389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.949200 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11379-3d498e" Oct 30 21:54:47.949389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.949253 4918 scope.go:122] "RemoveContainer" containerID="31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c" Oct 30 21:54:47.960176 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.960149 4918 scope.go:122] "RemoveContainer" containerID="31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c" Oct 30 21:54:47.960413 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:54:47.960393 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c\": container with ID starting with 31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c not found: ID does not exist" containerID="31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c" Oct 30 21:54:47.960466 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:47.960426 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c"} err="failed to get container status \"31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c\": rpc error: code = NotFound desc = could not find container \"31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c\": container with ID starting with 31d46594a760c3eb5139a6f5b524f6ff79f30bc10769d1945c6342f3d4efe09c not found: ID does not exist" Oct 30 21:54:52.493910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:52.493868 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.049916 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:57.049881 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.599192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:57.596827 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.796368 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:54:57.796335 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:02.153507 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:02.153472 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:02.402207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:02.402167 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:02.622894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:02.622863 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:07.041115 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:07.041074 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:08.924964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:08.924177 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": read tcp 10.132.0.2:45496->10.132.0.121:15021: read: connection reset by peer" start-of-body= Oct 30 21:55:08.924964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:08.924245 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": read tcp 10.132.0.2:45496->10.132.0.121:15021: read: connection reset by peer" Oct 30 21:55:09.184167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.184086 4918 generic.go:356] "Generic (PLEG): container finished" podID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerID="8decb4ed657a512d3a058ab20701254257788d1e234e463d301f0268b40a1f03" exitCode=0 Oct 30 21:55:09.184318 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.184166 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerDied","Data":"8decb4ed657a512d3a058ab20701254257788d1e234e463d301f0268b40a1f03"} Oct 30 21:55:09.184588 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.184572 4918 scope.go:122] "RemoveContainer" containerID="8decb4ed657a512d3a058ab20701254257788d1e234e463d301f0268b40a1f03" Oct 30 21:55:09.186373 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.186338 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:55:09.186493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.186427 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:55:09.186493 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:09.186478 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:55:10.214582 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:10.214492 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerStarted","Data":"94676b8f1836a4306e0ebccf37d15cd17504ead3378489effb7b91e3a3537519"} Oct 30 21:55:10.750784 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:10.750749 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:55:10.750982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:10.750950 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:55:10.751128 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:10.751017 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:55:11.721422 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:11.721386 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.750692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:11.750654 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:55:11.750692 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:11.750724 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:55:12.750801 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:12.750768 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:55:12.751190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:12.750824 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:55:13.750282 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:13.750247 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:55:13.750481 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:13.750313 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:55:14.751036 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:14.750997 4918 patch_prober.go:29] interesting pod/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:55:14.751521 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:14.751051 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:55:15.755475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:15.755442 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:55:15.755971 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:15.755954 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:55:15.756603 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:15.756584 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 21:55:15.806544 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:55:15.806509 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:55:16.375380 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:16.375333 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.848216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:20.848181 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.035865 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:21.035827 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.232512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:21.232478 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:25.653015 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:25.652955 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.441392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:30.441344 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:35.105986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:35.105935 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:35.324184 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:35.324141 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:35.523717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:35.523667 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:40.129894 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:40.129836 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:43.938064 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:43.938024 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:44.865732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:44.865664 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:45.803889 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:55:45.803858 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:55:49.465163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.465132 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:55:49.465988 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.465965 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" containerName="test-curl-11379-3d498e" Oct 30 21:55:49.465988 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.465984 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" containerName="test-curl-11379-3d498e" Oct 30 21:55:49.466145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.466088 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b517c8-106f-4fe3-b291-cbc3745fc5c0" containerName="test-curl-11379-3d498e" Oct 30 21:55:49.471207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.471185 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.566956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.566912 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-167561-serving-cert\"" Oct 30 21:55:49.573388 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.573358 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:55:49.579142 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.579098 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.579329 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.579297 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.579529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.579361 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jsr\" (UniqueName: \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.579529 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.579483 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.679555 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.679508 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:49.688443 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.688407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.692933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.691941 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.693269 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.692847 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.693405 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.692902 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.693732 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.693691 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.693931 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.693907 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jsr\" (UniqueName: \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.694202 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:55:49.693905 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-serving-cert: secret "maas-api-e2e-shared-b-w0-167561-serving-cert" not found Oct 30 21:55:49.695856 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:55:49.695840 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls podName:f2607bb4-a4b4-4809-9413-3fe1520516ea nodeName:}" failed. No retries permitted until 2026-10-30 21:55:50.19581724 +0000 UTC m=+2530.889299467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls") pod "maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" (UID: "f2607bb4-a4b4-4809-9413-3fe1520516ea") : secret "maas-api-e2e-shared-b-w0-167561-serving-cert" not found Oct 30 21:55:49.753163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.753069 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jsr\" (UniqueName: \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:49.936956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:49.936923 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:50.201228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:50.201193 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:50.203853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:50.203826 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:50.265425 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:50.265390 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:50.387162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:50.387129 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:50.782836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:50.782774 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:55:50.783998 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:55:50.783968 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2607bb4_a4b4_4809_9413_3fe1520516ea.slice/crio-d848d659345219ff47b37537a55f8333b76145e942d3d4161e5fd46488cd7375 WatchSource:0}: Error finding container d848d659345219ff47b37537a55f8333b76145e942d3d4161e5fd46488cd7375: Status 404 returned error can't find the container with id d848d659345219ff47b37537a55f8333b76145e942d3d4161e5fd46488cd7375 Oct 30 21:55:51.564669 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:51.564628 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" event={"ID":"f2607bb4-a4b4-4809-9413-3fe1520516ea","Type":"ContainerStarted","Data":"d848d659345219ff47b37537a55f8333b76145e942d3d4161e5fd46488cd7375"} Oct 30 21:55:52.581730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:52.577152 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" event={"ID":"f2607bb4-a4b4-4809-9413-3fe1520516ea","Type":"ContainerStarted","Data":"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27"} Oct 30 21:55:52.581730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:52.578317 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:55:52.697534 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:52.697483 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" podStartSLOduration=2.808360669 podStartE2EDuration="3.697468996s" totalImagesPullingTime="889.108327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:52.696565557 +0000 UTC m=+2533.390047793" watchObservedRunningTime="2026-10-30 21:55:52.697468996 +0000 UTC m=+2533.390951232" Oct 30 21:55:54.383748 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:54.378542 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:54.796484 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:54.794541 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.434180 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:55.434126 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.435949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:56.435886 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.656760 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:56.656726 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:59.602505 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:55:59.602475 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:56:00.040928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:00.040896 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.527250 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:00.527211 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.763041 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:00.763008 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.974303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:00.974262 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:03.901650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:03.901615 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:04.103345 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:04.103304 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:05.258030 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:05.257976 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:10.367095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:10.367035 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:15.451685 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:15.451651 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:15.812477 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:56:15.812389 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:56:17.858671 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:17.858635 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:19.415858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:19.415826 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:19.613095 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:19.613041 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:20.442166 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:20.442126 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:20.979763 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:20.979693 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:21.198144 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:21.198079 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:25.336983 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:25.336948 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:30.115451 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:30.115416 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:34.926573 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:34.926507 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:35.981617 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:35.981519 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:36.180689 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:36.180649 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.439986 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:37.439953 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.650898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:37.650859 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:38.010686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.010607 4918 generic.go:356] "Generic (PLEG): container finished" podID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerID="9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84" exitCode=0 Oct 30 21:56:38.010933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.010736 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerDied","Data":"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84"} Oct 30 21:56:38.011240 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.011223 4918 scope.go:122] "RemoveContainer" containerID="9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84" Oct 30 21:56:38.013274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.013247 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:56:38.013344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.013320 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:56:38.013390 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:38.013353 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:56:39.021577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.021540 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerStarted","Data":"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a"} Oct 30 21:56:39.301811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.301697 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:39.496774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.496738 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:56:39.498675 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.498642 4918 patch_prober.go:29] interesting pod/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:39.498852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.498698 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:56:39.580775 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:39.580436 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:56:40.079945 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:40.079908 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:40.496953 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:40.496918 4918 patch_prober.go:29] interesting pod/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:40.497129 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:40.496972 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:56:41.496863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:41.496825 4918 patch_prober.go:29] interesting pod/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:41.497253 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:41.496883 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:56:42.497458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:42.497422 4918 patch_prober.go:29] interesting pod/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:42.497934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:42.497482 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:56:43.497009 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:43.496970 4918 patch_prober.go:29] interesting pod/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:43.497287 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:43.497258 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:56:44.316260 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:44.316223 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.502049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:44.502017 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:56:44.503174 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:44.503150 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 21:56:44.508438 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:44.508409 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.826303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:44.826270 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:45.808750 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:56:45.808686 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:56:46.414431 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:46.414390 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:48.368167 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:48.368130 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:48.690720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:48.690631 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:49.541149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:49.541107 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:54.219653 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:54.219615 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.700284 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:56:59.700230 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:04.860779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:04.860737 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.840932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:09.840880 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:10.206551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:10.206513 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:13.579219 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:13.579184 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:57:13.686280 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:13.686246 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:57:14.005357 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:14.005318 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:14.228013 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:14.227975 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:15.217011 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:15.216972 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:15.804872 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:15.804837 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:57:17.450007 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:17.449977 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:18.579910 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.579872 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" podUID="c7393e77-09f5-406c-a781-4af580134e07" containerName="payload-processing" containerID="cri-o://88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469" gracePeriod=25 Oct 30 21:57:18.687465 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.687429 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" podUID="2220fea4-1ae3-4a51-b028-e912df0d4001" containerName="payload-pre-processing" containerID="cri-o://0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5" gracePeriod=25 Oct 30 21:57:18.885341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.885312 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:57:18.970088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.969247 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\") pod \"c7393e77-09f5-406c-a781-4af580134e07\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " Oct 30 21:57:18.970088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.969395 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\" (UniqueName: \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\") pod \"c7393e77-09f5-406c-a781-4af580134e07\" (UID: \"c7393e77-09f5-406c-a781-4af580134e07\") " Oct 30 21:57:18.970088 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.969743 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume" pod "c7393e77-09f5-406c-a781-4af580134e07" (UID: "c7393e77-09f5-406c-a781-4af580134e07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:18.970386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.970336 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7393e77-09f5-406c-a781-4af580134e07-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:18.974778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:18.974745 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5" pod "c7393e77-09f5-406c-a781-4af580134e07" (UID: "c7393e77-09f5-406c-a781-4af580134e07"). InnerVolumeSpecName "kube-api-access-wl8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.011363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.011338 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:57:19.071112 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.071086 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\") pod \"2220fea4-1ae3-4a51-b028-e912df0d4001\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " Oct 30 21:57:19.071305 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.071292 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\" (UniqueName: \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\") pod \"2220fea4-1ae3-4a51-b028-e912df0d4001\" (UID: \"2220fea4-1ae3-4a51-b028-e912df0d4001\") " Oct 30 21:57:19.071480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.071457 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume" pod "2220fea4-1ae3-4a51-b028-e912df0d4001" (UID: "2220fea4-1ae3-4a51-b028-e912df0d4001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:19.071776 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.071758 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8l5\" (UniqueName: \"kubernetes.io/projected/c7393e77-09f5-406c-a781-4af580134e07-kube-api-access-wl8l5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.071861 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.071779 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fea4-1ae3-4a51-b028-e912df0d4001-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.073317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.073293 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg" pod "2220fea4-1ae3-4a51-b028-e912df0d4001" (UID: "2220fea4-1ae3-4a51-b028-e912df0d4001"). InnerVolumeSpecName "kube-api-access-8d4zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.173067 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.173033 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4zg\" (UniqueName: \"kubernetes.io/projected/2220fea4-1ae3-4a51-b028-e912df0d4001-kube-api-access-8d4zg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.448272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.448243 4918 generic.go:356] "Generic (PLEG): container finished" podID="2220fea4-1ae3-4a51-b028-e912df0d4001" containerID="0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5" exitCode=0 Oct 30 21:57:19.448471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.448295 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" Oct 30 21:57:19.448471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.448334 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" event={"ID":"2220fea4-1ae3-4a51-b028-e912df0d4001","Type":"ContainerDied","Data":"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5"} Oct 30 21:57:19.448471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.448381 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt" event={"ID":"2220fea4-1ae3-4a51-b028-e912df0d4001","Type":"ContainerDied","Data":"3387acecfce00e2601a5df36fae776efca631806aecb0692170082ed7672b2e2"} Oct 30 21:57:19.448471 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.448411 4918 scope.go:122] "RemoveContainer" containerID="0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5" Oct 30 21:57:19.451203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.451180 4918 generic.go:356] "Generic (PLEG): container finished" podID="c7393e77-09f5-406c-a781-4af580134e07" containerID="88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469" exitCode=0 Oct 30 21:57:19.451316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.451206 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" event={"ID":"c7393e77-09f5-406c-a781-4af580134e07","Type":"ContainerDied","Data":"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469"} Oct 30 21:57:19.451316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.451233 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" Oct 30 21:57:19.451316 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.451250 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx" event={"ID":"c7393e77-09f5-406c-a781-4af580134e07","Type":"ContainerDied","Data":"98c49f1c6d648ede5ea607ab09bc7ec64e0955b9ac4a77247192f6e3cc1ae157"} Oct 30 21:57:19.462060 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.462043 4918 scope.go:122] "RemoveContainer" containerID="0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5" Oct 30 21:57:19.462335 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:19.462316 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5\": container with ID starting with 0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5 not found: ID does not exist" containerID="0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5" Oct 30 21:57:19.462410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.462346 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5"} err="failed to get container status \"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5\": rpc error: code = NotFound desc = could not find container \"0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5\": container with ID starting with 0418e1b85087d9dcded4f21f54f4514271dea09781a43dd0b27cba08460bdde5 not found: ID does not exist" Oct 30 21:57:19.462410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.462370 4918 scope.go:122] "RemoveContainer" containerID="88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469" Oct 30 21:57:19.472528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.472504 4918 scope.go:122] "RemoveContainer" containerID="88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469" Oct 30 21:57:19.472828 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:19.472809 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469\": container with ID starting with 88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469 not found: ID does not exist" containerID="88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469" Oct 30 21:57:19.472898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.472838 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469"} err="failed to get container status \"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469\": rpc error: code = NotFound desc = could not find container \"88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469\": container with ID starting with 88cfa468466147ccf85d7ccadd91c3fae483cad984860816a5b3b3b07fe38469 not found: ID does not exist" Oct 30 21:57:19.608921 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.608893 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:57:19.652837 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.652810 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d7f57-585d9fb4f9-phcmt"] Oct 30 21:57:19.756418 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.756336 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:57:19.762731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:19.762685 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d7f57-747866f95b-zfsbx"] Oct 30 21:57:20.045496 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:20.045406 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:21.433163 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:21.433128 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2220fea4-1ae3-4a51-b028-e912df0d4001" path="/var/lib/kubelet/pods/2220fea4-1ae3-4a51-b028-e912df0d4001/volumes" Oct 30 21:57:21.433682 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:21.433664 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7393e77-09f5-406c-a781-4af580134e07" path="/var/lib/kubelet/pods/c7393e77-09f5-406c-a781-4af580134e07/volumes" Oct 30 21:57:23.844961 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:23.844927 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:57:23.845423 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:23.845130 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" containerID="cri-o://c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702" gracePeriod=30 Oct 30 21:57:23.894627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:23.894590 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:57:23.894867 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:23.894825 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" containerID="cri-o://951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea" gracePeriod=30 Oct 30 21:57:24.069272 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:24.069238 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:57:24.180952 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:24.180921 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:57:24.492370 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:24.492268 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.708403 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:24.708358 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.932798 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:24.932763 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.066310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:25.066268 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:25.066465 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:25.066324 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:25.194080 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:25.193989 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:25.194080 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:25.194069 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:29.071724 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.070352 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" podUID="84f5b08c-08af-4128-a346-2880773b5047" containerName="payload-processing" containerID="cri-o://c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555" gracePeriod=25 Oct 30 21:57:29.184035 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.183990 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" podUID="ec34a0f9-2604-4ce0-897b-786f983d1fad" containerName="payload-pre-processing" containerID="cri-o://68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c" gracePeriod=25 Oct 30 21:57:29.259978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.259883 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:57:29.308828 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.308800 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:57:29.368274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.368243 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.430964 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431029 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431069 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431140 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\") pod \"84f5b08c-08af-4128-a346-2880773b5047\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431218 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431258 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431291 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431326 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431378 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431517 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\" (UniqueName: \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\") pod \"84f5b08c-08af-4128-a346-2880773b5047\" (UID: \"84f5b08c-08af-4128-a346-2880773b5047\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431551 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431584 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431628 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431658 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431700 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431807 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.431919 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431845 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.432892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431930 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\") pod \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\" (UID: \"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2\") " Oct 30 21:57:29.432892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.431981 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.432892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.432062 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\") pod \"cee57187-4974-4a5d-b0f8-301a109e8f04\" (UID: \"cee57187-4974-4a5d-b0f8-301a109e8f04\") " Oct 30 21:57:29.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.435873 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume" pod "84f5b08c-08af-4128-a346-2880773b5047" (UID: "84f5b08c-08af-4128-a346-2880773b5047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:29.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.436720 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.437045 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:29.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.437292 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.437787 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.437497 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.438459 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.438228 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.438940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.438644 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.441149 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.441106 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.445046 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7" pod "84f5b08c-08af-4128-a346-2880773b5047" (UID: "84f5b08c-08af-4128-a346-2880773b5047"). InnerVolumeSpecName "kube-api-access-f95m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.445393 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.445649 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.445861 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "kube-api-access-zpqfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.446778 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.447024 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.447211 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.447324 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.447829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.447579 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:29.449109 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.449069 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token" pod "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" (UID: "7cdb39ff-d4e2-4023-a65e-c15faa8d54a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.449208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.449103 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "kube-api-access-8dhzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.450465 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.450444 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo" pod "cee57187-4974-4a5d-b0f8-301a109e8f04" (UID: "cee57187-4974-4a5d-b0f8-301a109e8f04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:29.517552 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.517483 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.537226 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\") pod \"ec34a0f9-2604-4ce0-897b-786f983d1fad\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.537409 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\" (UniqueName: \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\") pod \"ec34a0f9-2604-4ce0-897b-786f983d1fad\" (UID: \"ec34a0f9-2604-4ce0-897b-786f983d1fad\") " Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538135 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhzw\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-kube-api-access-8dhzw\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538160 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538174 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee57187-4974-4a5d-b0f8-301a109e8f04-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538187 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f5b08c-08af-4128-a346-2880773b5047-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538199 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538211 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538225 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538237 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538249 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538261 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f95m7\" (UniqueName: \"kubernetes.io/projected/84f5b08c-08af-4128-a346-2880773b5047-kube-api-access-f95m7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538275 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538288 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538302 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538314 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538329 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538341 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqfh\" (UniqueName: \"kubernetes.io/projected/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-kube-api-access-zpqfh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538353 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538381 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538394 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee57187-4974-4a5d-b0f8-301a109e8f04-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.538407 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee57187-4974-4a5d-b0f8-301a109e8f04-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.537402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.539683 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume" pod "ec34a0f9-2604-4ce0-897b-786f983d1fad" (UID: "ec34a0f9-2604-4ce0-897b-786f983d1fad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:29.543823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.543649 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh" pod "ec34a0f9-2604-4ce0-897b-786f983d1fad" (UID: "ec34a0f9-2604-4ce0-897b-786f983d1fad"). InnerVolumeSpecName "kube-api-access-tj8zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:29.581096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.581064 4918 generic.go:356] "Generic (PLEG): container finished" podID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerID="951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea" exitCode=0 Oct 30 21:57:29.581699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.581282 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" event={"ID":"cee57187-4974-4a5d-b0f8-301a109e8f04","Type":"ContainerDied","Data":"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea"} Oct 30 21:57:29.581869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.581793 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" event={"ID":"cee57187-4974-4a5d-b0f8-301a109e8f04","Type":"ContainerDied","Data":"7e76d0d4140c4786269d34ff7fe258abd28cf5e3413a92f042c49ae6b11c3342"} Oct 30 21:57:29.581869 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.581841 4918 scope.go:122] "RemoveContainer" containerID="951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea" Oct 30 21:57:29.582057 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.581345 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t" Oct 30 21:57:29.586923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.586001 4918 generic.go:356] "Generic (PLEG): container finished" podID="84f5b08c-08af-4128-a346-2880773b5047" containerID="c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555" exitCode=0 Oct 30 21:57:29.586923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.586107 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" event={"ID":"84f5b08c-08af-4128-a346-2880773b5047","Type":"ContainerDied","Data":"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555"} Oct 30 21:57:29.586923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.586149 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" event={"ID":"84f5b08c-08af-4128-a346-2880773b5047","Type":"ContainerDied","Data":"0375f1e9b03672a059fd200d8dc92af43da695fc477b99ca2bbdbcca8033a64f"} Oct 30 21:57:29.586923 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.586220 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf" Oct 30 21:57:29.590577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.589647 4918 generic.go:356] "Generic (PLEG): container finished" podID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerID="c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702" exitCode=0 Oct 30 21:57:29.590577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.589763 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" event={"ID":"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2","Type":"ContainerDied","Data":"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702"} Oct 30 21:57:29.590577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.589803 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" event={"ID":"7cdb39ff-d4e2-4023-a65e-c15faa8d54a2","Type":"ContainerDied","Data":"54487c22633f40d4a7380f3df39f78b9af8c651d2a1260041f431b588373997e"} Oct 30 21:57:29.590577 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.589880 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h" Oct 30 21:57:29.593454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.593432 4918 generic.go:356] "Generic (PLEG): container finished" podID="ec34a0f9-2604-4ce0-897b-786f983d1fad" containerID="68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c" exitCode=0 Oct 30 21:57:29.593583 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.593568 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" event={"ID":"ec34a0f9-2604-4ce0-897b-786f983d1fad","Type":"ContainerDied","Data":"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c"} Oct 30 21:57:29.593721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.593607 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" event={"ID":"ec34a0f9-2604-4ce0-897b-786f983d1fad","Type":"ContainerDied","Data":"c9ba5547121be600cc66f6e53959a16ed56fd072cde0df03efcc7fc896048a0e"} Oct 30 21:57:29.593721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.593674 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm" Oct 30 21:57:29.609407 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.609388 4918 scope.go:122] "RemoveContainer" containerID="951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea" Oct 30 21:57:29.609740 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:29.609694 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea\": container with ID starting with 951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea not found: ID does not exist" containerID="951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea" Oct 30 21:57:29.609853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.609742 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea"} err="failed to get container status \"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea\": rpc error: code = NotFound desc = could not find container \"951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea\": container with ID starting with 951a169e25171fd06d90d788234d564031439945f7381c4af08d3ec40716faea not found: ID does not exist" Oct 30 21:57:29.609853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.609765 4918 scope.go:122] "RemoveContainer" containerID="c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555" Oct 30 21:57:29.619935 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.619911 4918 scope.go:122] "RemoveContainer" containerID="c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555" Oct 30 21:57:29.620220 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:29.620195 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555\": container with ID starting with c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555 not found: ID does not exist" containerID="c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555" Oct 30 21:57:29.620291 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.620230 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555"} err="failed to get container status \"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555\": rpc error: code = NotFound desc = could not find container \"c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555\": container with ID starting with c0424a4f7cb6ea81008d8225167d145fe7a23fa99559ab40cfd3d44b7077d555 not found: ID does not exist" Oct 30 21:57:29.620291 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.620251 4918 scope.go:122] "RemoveContainer" containerID="c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702" Oct 30 21:57:29.629876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.629859 4918 scope.go:122] "RemoveContainer" containerID="c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702" Oct 30 21:57:29.630147 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:29.630130 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702\": container with ID starting with c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702 not found: ID does not exist" containerID="c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702" Oct 30 21:57:29.630236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.630156 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702"} err="failed to get container status \"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702\": rpc error: code = NotFound desc = could not find container \"c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702\": container with ID starting with c1a740e2d9b2588ec839f9814586bf20f0994fcfab0849c0266ea543bd5c5702 not found: ID does not exist" Oct 30 21:57:29.630236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.630179 4918 scope.go:122] "RemoveContainer" containerID="68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c" Oct 30 21:57:29.639876 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.639853 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8zh\" (UniqueName: \"kubernetes.io/projected/ec34a0f9-2604-4ce0-897b-786f983d1fad-kube-api-access-tj8zh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.639966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.639881 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec34a0f9-2604-4ce0-897b-786f983d1fad-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.639966 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.639904 4918 scope.go:122] "RemoveContainer" containerID="68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c" Oct 30 21:57:29.640180 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:29.640163 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c\": container with ID starting with 68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c not found: ID does not exist" containerID="68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c" Oct 30 21:57:29.640236 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.640191 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c"} err="failed to get container status \"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c\": rpc error: code = NotFound desc = could not find container \"68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c\": container with ID starting with 68b350f8168bf453827495b82a7fd8670e86a42ddb43ad9958fc75c0fd6c123c not found: ID does not exist" Oct 30 21:57:29.766026 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.765940 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:57:29.772127 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.772099 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-737a64-68c9b4cd7d-ffwvf"] Oct 30 21:57:29.860909 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.860884 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 21:57:29.861652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861634 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c7393e77-09f5-406c-a781-4af580134e07" containerName="payload-processing" Oct 30 21:57:29.861652 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861651 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7393e77-09f5-406c-a781-4af580134e07" containerName="payload-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861674 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861679 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861688 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ec34a0f9-2604-4ce0-897b-786f983d1fad" containerName="payload-pre-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861693 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec34a0f9-2604-4ce0-897b-786f983d1fad" containerName="payload-pre-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861740 4918 cpu_manager.go:414] "containerMap: removing container" podUID="84f5b08c-08af-4128-a346-2880773b5047" containerName="payload-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861746 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f5b08c-08af-4128-a346-2880773b5047" containerName="payload-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861761 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861769 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861793 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2220fea4-1ae3-4a51-b028-e912df0d4001" containerName="payload-pre-processing" Oct 30 21:57:29.861807 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861798 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2220fea4-1ae3-4a51-b028-e912df0d4001" containerName="payload-pre-processing" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861934 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2220fea4-1ae3-4a51-b028-e912df0d4001" containerName="payload-pre-processing" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861946 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7393e77-09f5-406c-a781-4af580134e07" containerName="payload-processing" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861957 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" containerName="istio-proxy" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861964 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec34a0f9-2604-4ce0-897b-786f983d1fad" containerName="payload-pre-processing" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861974 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f5b08c-08af-4128-a346-2880773b5047" containerName="payload-processing" Oct 30 21:57:29.862120 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.861980 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" containerName="istio-proxy" Oct 30 21:57:29.867190 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.867172 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:29.916830 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.916669 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c4e874-serving-cert\"" Oct 30 21:57:29.923806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.923785 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 21:57:29.943626 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.943599 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6gx\" (UniqueName: \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:29.943932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.943910 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:29.944159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.944139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:29.944342 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.944319 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:29.944512 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.944495 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:57:29.949203 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:29.949182 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-737a64-5f4f7c555f-rqsjm"] Oct 30 21:57:30.045305 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.045214 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.045569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.045446 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6gx\" (UniqueName: \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.045569 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.045553 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.045802 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.045588 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.046344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:30.046139 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-serving-cert: secret "maas-api-e2e-api-b-c4e874-serving-cert" not found Oct 30 21:57:30.046344 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:30.046210 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls podName:2aeec501-49d9-44a1-bb78-ae7ac11a65a2 nodeName:}" failed. No retries permitted until 2026-10-30 21:57:30.546188069 +0000 UTC m=+2631.239670297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls") pod "maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" (UID: "2aeec501-49d9-44a1-bb78-ae7ac11a65a2") : secret "maas-api-e2e-api-b-c4e874-serving-cert" not found Oct 30 21:57:30.046559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.046410 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.049210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.049181 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.051631 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.051607 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:57:30.058478 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.058455 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-mz82t"] Oct 30 21:57:30.122649 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.122614 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6gx\" (UniqueName: \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.140296 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.140265 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.166815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.166764 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:57:30.191772 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.190397 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2pr6h"] Oct 30 21:57:30.366932 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.366904 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.553608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.553575 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.556037 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.556015 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e874-66c758b5f-pqtft\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:30.580252 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.580217 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.824845 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:30.824768 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:31.218779 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:57:31.215174 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aeec501_49d9_44a1_bb78_ae7ac11a65a2.slice/crio-00834e83fa32f5e761a5205eb7e735def7cee2a2cd5fc7007d82fd755a63c920 WatchSource:0}: Error finding container 00834e83fa32f5e761a5205eb7e735def7cee2a2cd5fc7007d82fd755a63c920: Status 404 returned error can't find the container with id 00834e83fa32f5e761a5205eb7e735def7cee2a2cd5fc7007d82fd755a63c920 Oct 30 21:57:31.218779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.215499 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 21:57:31.218779 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.217001 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:57:31.430341 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.430308 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdb39ff-d4e2-4023-a65e-c15faa8d54a2" path="/var/lib/kubelet/pods/7cdb39ff-d4e2-4023-a65e-c15faa8d54a2/volumes" Oct 30 21:57:31.430766 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.430750 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f5b08c-08af-4128-a346-2880773b5047" path="/var/lib/kubelet/pods/84f5b08c-08af-4128-a346-2880773b5047/volumes" Oct 30 21:57:31.431207 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.431194 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee57187-4974-4a5d-b0f8-301a109e8f04" path="/var/lib/kubelet/pods/cee57187-4974-4a5d-b0f8-301a109e8f04/volumes" Oct 30 21:57:31.431933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.431918 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec34a0f9-2604-4ce0-897b-786f983d1fad" path="/var/lib/kubelet/pods/ec34a0f9-2604-4ce0-897b-786f983d1fad/volumes" Oct 30 21:57:31.628323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:31.628290 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" event={"ID":"2aeec501-49d9-44a1-bb78-ae7ac11a65a2","Type":"ContainerStarted","Data":"00834e83fa32f5e761a5205eb7e735def7cee2a2cd5fc7007d82fd755a63c920"} Oct 30 21:57:32.637439 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:32.637406 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" event={"ID":"2aeec501-49d9-44a1-bb78-ae7ac11a65a2","Type":"ContainerStarted","Data":"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9"} Oct 30 21:57:32.637892 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:32.637465 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:32.790159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:32.790100 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" podStartSLOduration=2.976852072 podStartE2EDuration="3.790085179s" totalImagesPullingTime="813.233107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:32.78509937 +0000 UTC m=+2633.478581607" watchObservedRunningTime="2026-10-30 21:57:32.790085179 +0000 UTC m=+2633.483567446" Oct 30 21:57:33.410208 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:33.410177 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:33.645940 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:33.645908 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:57:33.703778 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:33.703690 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:33.716152 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:33.716129 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:57:34.063844 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:34.063487 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.389811 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:34.389777 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:35.734363 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:35.734321 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:37.283205 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:37.283174 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:38.646855 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:38.646815 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" podUID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" containerName="payload-processing" containerID="cri-o://1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354" gracePeriod=25 Oct 30 21:57:38.651427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:38.651321 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 21:57:38.717249 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:38.717221 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" podUID="c28f5ed2-9120-4340-9eb2-b282ee98d733" containerName="payload-pre-processing" containerID="cri-o://781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163" gracePeriod=25 Oct 30 21:57:39.047539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.047514 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:57:39.067145 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.067121 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:57:39.180079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180053 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\" (UniqueName: \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\") pod \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " Oct 30 21:57:39.180229 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180130 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\") pod \"c28f5ed2-9120-4340-9eb2-b282ee98d733\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " Oct 30 21:57:39.180321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180303 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\") pod \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\" (UID: \"0bde51c2-d7cb-47c0-be0b-018e87d3449a\") " Oct 30 21:57:39.180741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180659 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume" pod "c28f5ed2-9120-4340-9eb2-b282ee98d733" (UID: "c28f5ed2-9120-4340-9eb2-b282ee98d733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:39.180741 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180690 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume" pod "0bde51c2-d7cb-47c0-be0b-018e87d3449a" (UID: "0bde51c2-d7cb-47c0-be0b-018e87d3449a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:39.180922 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.180889 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\" (UniqueName: \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\") pod \"c28f5ed2-9120-4340-9eb2-b282ee98d733\" (UID: \"c28f5ed2-9120-4340-9eb2-b282ee98d733\") " Oct 30 21:57:39.181449 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.181425 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28f5ed2-9120-4340-9eb2-b282ee98d733-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:39.181560 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.181452 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bde51c2-d7cb-47c0-be0b-018e87d3449a-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:39.182882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.182862 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66" pod "0bde51c2-d7cb-47c0-be0b-018e87d3449a" (UID: "0bde51c2-d7cb-47c0-be0b-018e87d3449a"). InnerVolumeSpecName "kube-api-access-f8c66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:39.183253 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.183231 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz" pod "c28f5ed2-9120-4340-9eb2-b282ee98d733" (UID: "c28f5ed2-9120-4340-9eb2-b282ee98d733"). InnerVolumeSpecName "kube-api-access-kwfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:39.283029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.282999 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfzz\" (UniqueName: \"kubernetes.io/projected/c28f5ed2-9120-4340-9eb2-b282ee98d733-kube-api-access-kwfzz\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:39.283131 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.283033 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8c66\" (UniqueName: \"kubernetes.io/projected/0bde51c2-d7cb-47c0-be0b-018e87d3449a-kube-api-access-f8c66\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:39.596404 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.596360 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:39.697788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.697691 4918 generic.go:356] "Generic (PLEG): container finished" podID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" containerID="1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354" exitCode=0 Oct 30 21:57:39.697788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.697762 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" Oct 30 21:57:39.697788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.697774 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" event={"ID":"0bde51c2-d7cb-47c0-be0b-018e87d3449a","Type":"ContainerDied","Data":"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354"} Oct 30 21:57:39.698317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.697831 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv" event={"ID":"0bde51c2-d7cb-47c0-be0b-018e87d3449a","Type":"ContainerDied","Data":"cdd87d070cea6d2538e88027ffb196254b27287a16d943851feb927dbcee9fe5"} Oct 30 21:57:39.698317 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.697863 4918 scope.go:122] "RemoveContainer" containerID="1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354" Oct 30 21:57:39.699853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.699829 4918 generic.go:356] "Generic (PLEG): container finished" podID="c28f5ed2-9120-4340-9eb2-b282ee98d733" containerID="781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163" exitCode=0 Oct 30 21:57:39.699967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.699937 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" event={"ID":"c28f5ed2-9120-4340-9eb2-b282ee98d733","Type":"ContainerDied","Data":"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163"} Oct 30 21:57:39.700082 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.699988 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" event={"ID":"c28f5ed2-9120-4340-9eb2-b282ee98d733","Type":"ContainerDied","Data":"31f0f64b4b69ffd561ab257c0cab6e345c02ec64273c0c70437f6b1ebe67e25c"} Oct 30 21:57:39.700446 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.700424 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc" Oct 30 21:57:39.709490 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.709473 4918 scope.go:122] "RemoveContainer" containerID="1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354" Oct 30 21:57:39.709769 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:39.709747 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354\": container with ID starting with 1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354 not found: ID does not exist" containerID="1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354" Oct 30 21:57:39.709856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.709779 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354"} err="failed to get container status \"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354\": rpc error: code = NotFound desc = could not find container \"1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354\": container with ID starting with 1d93c5a13b9863cfb257386378c013886441c6a5ecee2753e4a9d34c0696c354 not found: ID does not exist" Oct 30 21:57:39.709856 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.709800 4918 scope.go:122] "RemoveContainer" containerID="781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163" Oct 30 21:57:39.719829 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.719809 4918 scope.go:122] "RemoveContainer" containerID="781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163" Oct 30 21:57:39.720102 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:39.720086 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163\": container with ID starting with 781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163 not found: ID does not exist" containerID="781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163" Oct 30 21:57:39.720162 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.720109 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163"} err="failed to get container status \"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163\": rpc error: code = NotFound desc = could not find container \"781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163\": container with ID starting with 781ae1b264afcd6b03ecef30ee575de98b01b1cd0037823e34bf41600720d163 not found: ID does not exist" Oct 30 21:57:39.855608 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.855579 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:57:39.864023 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.863997 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2aa268-564c46b759-dfpdc"] Oct 30 21:57:39.964831 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.964764 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:57:39.969029 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:39.969004 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-2aa268-74f986c584-f7lzv"] Oct 30 21:57:40.629427 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:40.629394 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:41.429579 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:41.429547 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" path="/var/lib/kubelet/pods/0bde51c2-d7cb-47c0-be0b-018e87d3449a/volumes" Oct 30 21:57:41.430010 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:41.429994 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28f5ed2-9120-4340-9eb2-b282ee98d733" path="/var/lib/kubelet/pods/c28f5ed2-9120-4340-9eb2-b282ee98d733/volumes" Oct 30 21:57:44.614327 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:44.614290 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:45.630441 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:45.630407 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:45.803172 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:57:45.803134 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:57:49.661323 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:49.661253 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.129815 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:51.129781 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.375381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:51.371335 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.705021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:51.704984 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:56.280069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:56.280029 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:59.969774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:57:59.969691 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:01.151159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:01.151119 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:03.776420 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:03.776389 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:03.964721 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:03.964673 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:06.116151 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:06.116116 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:06.322159 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:06.322119 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:11.355936 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:11.355900 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:15.803822 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:15.803781 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:58:16.249224 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:16.249183 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.497334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:20.497295 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.959792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:20.959745 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:21.995774 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:21.995731 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.280077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:22.279981 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.483258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:22.483214 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.906108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:22.906074 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:26.668228 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:26.668191 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:58:26.668638 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:26.668414 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" podUID="42889823-0309-4e14-924a-d0a673eb4ac0" containerName="maas-api" containerID="cri-o://1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551" gracePeriod=30 Oct 30 21:58:26.800156 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:26.800124 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:26.998788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:26.998761 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:58:27.095610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.095577 4918 generic.go:356] "Generic (PLEG): container finished" podID="42889823-0309-4e14-924a-d0a673eb4ac0" containerID="1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551" exitCode=0 Oct 30 21:58:27.095788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.095623 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" Oct 30 21:58:27.095788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.095657 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" event={"ID":"42889823-0309-4e14-924a-d0a673eb4ac0","Type":"ContainerDied","Data":"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551"} Oct 30 21:58:27.095788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.095726 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb" event={"ID":"42889823-0309-4e14-924a-d0a673eb4ac0","Type":"ContainerDied","Data":"6ca7b745f6ef96c66387dd1664ca3f5e6a978d6a97d9e75cacc8f66bfc4f28fd"} Oct 30 21:58:27.095788 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.095760 4918 scope.go:122] "RemoveContainer" containerID="1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551" Oct 30 21:58:27.107458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.107437 4918 scope.go:122] "RemoveContainer" containerID="1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551" Oct 30 21:58:27.110104 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:27.107806 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551\": container with ID starting with 1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551 not found: ID does not exist" containerID="1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551" Oct 30 21:58:27.110104 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.107841 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551"} err="failed to get container status \"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551\": rpc error: code = NotFound desc = could not find container \"1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551\": container with ID starting with 1f2fd0f2c02aad76622b6b4b08b1a366958c1a09b8576dfab8ceabcd2c789551 not found: ID does not exist" Oct 30 21:58:27.142482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.138993 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\") pod \"42889823-0309-4e14-924a-d0a673eb4ac0\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " Oct 30 21:58:27.142482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.139122 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\") pod \"42889823-0309-4e14-924a-d0a673eb4ac0\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " Oct 30 21:58:27.142482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.139202 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\") pod \"42889823-0309-4e14-924a-d0a673eb4ac0\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " Oct 30 21:58:27.142482 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.139266 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\" (UniqueName: \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\") pod \"42889823-0309-4e14-924a-d0a673eb4ac0\" (UID: \"42889823-0309-4e14-924a-d0a673eb4ac0\") " Oct 30 21:58:27.154434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.142963 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n" pod "42889823-0309-4e14-924a-d0a673eb4ac0" (UID: "42889823-0309-4e14-924a-d0a673eb4ac0"). InnerVolumeSpecName "kube-api-access-btk6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:27.154434 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.144121 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca" pod "42889823-0309-4e14-924a-d0a673eb4ac0" (UID: "42889823-0309-4e14-924a-d0a673eb4ac0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:27.162951 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.162915 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls" pod "42889823-0309-4e14-924a-d0a673eb4ac0" (UID: "42889823-0309-4e14-924a-d0a673eb4ac0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:27.181283 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.181232 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs" pod "42889823-0309-4e14-924a-d0a673eb4ac0" (UID: "42889823-0309-4e14-924a-d0a673eb4ac0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:27.244782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.244230 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:27.244782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.244271 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42889823-0309-4e14-924a-d0a673eb4ac0-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:27.244782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.244287 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42889823-0309-4e14-924a-d0a673eb4ac0-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:27.244782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.244303 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btk6n\" (UniqueName: \"kubernetes.io/projected/42889823-0309-4e14-924a-d0a673eb4ac0-kube-api-access-btk6n\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:27.521474 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.521402 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:58:27.528799 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:27.528775 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2aa268-6688ffbc56-t56fb"] Oct 30 21:58:28.509256 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:28.509221 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.731402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:28.731367 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.926348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:28.926311 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:29.186891 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:29.186802 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:29.431276 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:29.431241 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42889823-0309-4e14-924a-d0a673eb4ac0" path="/var/lib/kubelet/pods/42889823-0309-4e14-924a-d0a673eb4ac0/volumes" Oct 30 21:58:29.497217 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:29.497126 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.134304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:30.134260 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.328946 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:30.328915 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.153022 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:31.152955 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.351536 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:31.351502 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.556172 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:31.556096 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:33.413993 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:33.413959 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:34.044054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:34.044021 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.431647 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:37.431611 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.634635 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:37.634601 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.833304 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:37.833226 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:38.353574 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:38.353541 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:58:38.571358 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:38.571320 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.224883 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.224822 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.862765 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.862738 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:58:39.877792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.877765 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 21:58:39.877956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.877882 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:58:39.883720 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.883685 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:58:39.897630 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.897606 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 21:58:39.902868 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:39.902851 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 21:58:44.624688 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:44.624609 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:44.946765 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:44.946653 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:45.317192 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:45.317088 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:45.808463 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:45.808428 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:58:45.920193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:45.920160 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.129362 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:46.129330 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:49.716658 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:49.716622 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:50.287559 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:50.287523 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:58:50.388098 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:50.388069 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:58:50.768960 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:50.768926 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:50.978771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:50.978690 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.865839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:52.865807 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:58:52.977963 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:52.977927 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:58:53.301612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.301509 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:53.375294 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.375250 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" containerID="cri-o://925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1" gracePeriod=45 Oct 30 21:58:53.527387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.527347 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": dial tcp 10.132.0.89:8000: connect: connection refused" Oct 30 21:58:53.538970 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.538938 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:53.678691 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.678659 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:58:53.692982 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.692960 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\" (UniqueName: \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693014 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693108 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693095 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693169 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693314 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693300 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693377 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\") pod \"e008f318-954f-498c-8fc2-e638bfa5bba9\" (UID: \"e008f318-954f-498c-8fc2-e638bfa5bba9\") " Oct 30 21:58:53.693432 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693417 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.693557 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693531 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.693686 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693574 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.693978 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693954 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.694077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.693984 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.694077 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.694001 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.695852 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.695826 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:53.695955 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.695931 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.695997 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.695980 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m" pod "e008f318-954f-498c-8fc2-e638bfa5bba9" (UID: "e008f318-954f-498c-8fc2-e638bfa5bba9"). InnerVolumeSpecName "kube-api-access-mfm4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:53.794898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.794874 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e008f318-954f-498c-8fc2-e638bfa5bba9-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.794898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.794897 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e008f318-954f-498c-8fc2-e638bfa5bba9-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.795066 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:53.794908 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm4m\" (UniqueName: \"kubernetes.io/projected/e008f318-954f-498c-8fc2-e638bfa5bba9-kube-api-access-mfm4m\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:54.341428 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.341398 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" Oct 30 21:58:54.342054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.341431 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" event={"ID":"e008f318-954f-498c-8fc2-e638bfa5bba9","Type":"ContainerDied","Data":"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1"} Oct 30 21:58:54.342054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.341486 4918 scope.go:122] "RemoveContainer" containerID="925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1" Oct 30 21:58:54.342054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.341357 4918 generic.go:356] "Generic (PLEG): container finished" podID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerID="925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1" exitCode=0 Oct 30 21:58:54.342054 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.341596 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x" event={"ID":"e008f318-954f-498c-8fc2-e638bfa5bba9","Type":"ContainerDied","Data":"af7fde26497c3ef26ea2a552872d0eed499f4bdb8d6d300bce769144f9e17973"} Oct 30 21:58:54.354050 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.354033 4918 scope.go:122] "RemoveContainer" containerID="925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1" Oct 30 21:58:54.354337 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:54.354319 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1\": container with ID starting with 925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1 not found: ID does not exist" containerID="925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1" Oct 30 21:58:54.354389 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.354344 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1"} err="failed to get container status \"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1\": rpc error: code = NotFound desc = could not find container \"925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1\": container with ID starting with 925c93f708485fb275d8ccc0c4182cdf164843e69786a6811201f628e888bfc1 not found: ID does not exist" Oct 30 21:58:54.523532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.523500 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:58:54.534719 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.534680 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/distinct-2-w4-554f46-kserve-85667b8896-skm2x"] Oct 30 21:58:54.586353 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:54.586323 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:55.288665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.288627 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" podUID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" containerName="payload-processing" containerID="cri-o://df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856" gracePeriod=25 Oct 30 21:58:55.388392 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.388321 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" podUID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" containerName="payload-pre-processing" containerID="cri-o://60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567" gracePeriod=25 Oct 30 21:58:55.429780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.429752 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" path="/var/lib/kubelet/pods/e008f318-954f-498c-8fc2-e638bfa5bba9/volumes" Oct 30 21:58:55.590348 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.590316 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:58:55.623230 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.623190 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\") pod \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " Oct 30 21:58:55.623452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.623289 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\" (UniqueName: \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\") pod \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\" (UID: \"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5\") " Oct 30 21:58:55.623678 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.623650 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume" pod "ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" (UID: "ee5b3991-546f-4cdd-ad4b-1d80657ab6c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:55.624049 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.624023 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.634204 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.634166 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct" pod "ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" (UID: "ee5b3991-546f-4cdd-ad4b-1d80657ab6c5"). InnerVolumeSpecName "kube-api-access-bs9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:55.661943 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.661671 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:58:55.724670 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.724629 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\") pod \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " Oct 30 21:58:55.724959 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.724935 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\" (UniqueName: \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\") pod \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\" (UID: \"e00e074a-962f-439a-afc6-8ba3c3abf1d3\") " Oct 30 21:58:55.725433 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.725154 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume" pod "e00e074a-962f-439a-afc6-8ba3c3abf1d3" (UID: "e00e074a-962f-439a-afc6-8ba3c3abf1d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:55.725589 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.725569 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e00e074a-962f-439a-afc6-8ba3c3abf1d3-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.725660 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.725594 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9ct\" (UniqueName: \"kubernetes.io/projected/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5-kube-api-access-bs9ct\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.727898 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.727856 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b" pod "e00e074a-962f-439a-afc6-8ba3c3abf1d3" (UID: "e00e074a-962f-439a-afc6-8ba3c3abf1d3"). InnerVolumeSpecName "kube-api-access-7ht2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:55.827274 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:55.827181 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ht2b\" (UniqueName: \"kubernetes.io/projected/e00e074a-962f-439a-afc6-8ba3c3abf1d3-kube-api-access-7ht2b\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:56.363699 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.363664 4918 generic.go:356] "Generic (PLEG): container finished" podID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" containerID="60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567" exitCode=0 Oct 30 21:58:56.363884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.363726 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" Oct 30 21:58:56.363884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.363739 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" event={"ID":"e00e074a-962f-439a-afc6-8ba3c3abf1d3","Type":"ContainerDied","Data":"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567"} Oct 30 21:58:56.363884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.363794 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8" event={"ID":"e00e074a-962f-439a-afc6-8ba3c3abf1d3","Type":"ContainerDied","Data":"eb12ca48b101c7ebd8f6031134eef3d9dd85e0dc852b76652ad7ce964674496b"} Oct 30 21:58:56.363884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.363816 4918 scope.go:122] "RemoveContainer" containerID="60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567" Oct 30 21:58:56.365531 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.365506 4918 generic.go:356] "Generic (PLEG): container finished" podID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" containerID="df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856" exitCode=0 Oct 30 21:58:56.365764 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.365742 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" event={"ID":"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5","Type":"ContainerDied","Data":"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856"} Oct 30 21:58:56.365912 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.365785 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" event={"ID":"ee5b3991-546f-4cdd-ad4b-1d80657ab6c5","Type":"ContainerDied","Data":"0298b5243950f9138c9b42a8ffdfd5999c67690c66a7afc97db5e28b3983846b"} Oct 30 21:58:56.365912 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.365813 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8" Oct 30 21:58:56.454321 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.454296 4918 scope.go:122] "RemoveContainer" containerID="60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567" Oct 30 21:58:56.457243 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:56.457218 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567\": container with ID starting with 60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567 not found: ID does not exist" containerID="60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567" Oct 30 21:58:56.457387 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.457358 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567"} err="failed to get container status \"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567\": rpc error: code = NotFound desc = could not find container \"60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567\": container with ID starting with 60a4142efa5cb0f331f745b18d7206dafe887b2d00992b245fcea7979a4e6567 not found: ID does not exist" Oct 30 21:58:56.457492 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.457389 4918 scope.go:122] "RemoveContainer" containerID="df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856" Oct 30 21:58:56.522210 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.522114 4918 scope.go:122] "RemoveContainer" containerID="df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856" Oct 30 21:58:56.526727 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:56.525053 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856\": container with ID starting with df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856 not found: ID does not exist" containerID="df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856" Oct 30 21:58:56.526727 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.525097 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856"} err="failed to get container status \"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856\": rpc error: code = NotFound desc = could not find container \"df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856\": container with ID starting with df63eee0d4e476ab89d60ba81e17945713958c01123efcb146f6a799b7911856 not found: ID does not exist" Oct 30 21:58:56.648410 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.648316 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:58:56.664693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.664656 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-167561-5bf76f95f9-n4kk8"] Oct 30 21:58:56.742729 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.741753 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:58:56.750069 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:56.750041 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-167561-b744d8d86-txfq8"] Oct 30 21:58:57.453021 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:57.452984 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" path="/var/lib/kubelet/pods/e00e074a-962f-439a-afc6-8ba3c3abf1d3/volumes" Oct 30 21:58:57.453904 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:57.453882 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" path="/var/lib/kubelet/pods/ee5b3991-546f-4cdd-ad4b-1d80657ab6c5/volumes" Oct 30 21:58:57.866279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:57.866244 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" podUID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" containerName="payload-processing" containerID="cri-o://ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4" gracePeriod=25 Oct 30 21:58:57.978665 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:57.978631 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" podUID="471f3883-ac0a-4781-90e5-77dd38f6fffe" containerName="payload-pre-processing" containerID="cri-o://a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294" gracePeriod=25 Oct 30 21:58:58.164853 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.164729 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:58:58.283420 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.283391 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:58:58.290414 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.290388 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\") pod \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " Oct 30 21:58:58.290582 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.290442 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\" (UniqueName: \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\") pod \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\" (UID: \"36f0d01f-16c8-42ce-aa34-e0242d6bc53c\") " Oct 30 21:58:58.291472 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.291439 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume" pod "36f0d01f-16c8-42ce-aa34-e0242d6bc53c" (UID: "36f0d01f-16c8-42ce-aa34-e0242d6bc53c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:58.293045 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.293015 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7" pod "36f0d01f-16c8-42ce-aa34-e0242d6bc53c" (UID: "36f0d01f-16c8-42ce-aa34-e0242d6bc53c"). InnerVolumeSpecName "kube-api-access-lm6l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.392544 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\") pod \"471f3883-ac0a-4781-90e5-77dd38f6fffe\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.392752 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\" (UniqueName: \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\") pod \"471f3883-ac0a-4781-90e5-77dd38f6fffe\" (UID: \"471f3883-ac0a-4781-90e5-77dd38f6fffe\") " Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.392980 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume" pod "471f3883-ac0a-4781-90e5-77dd38f6fffe" (UID: "471f3883-ac0a-4781-90e5-77dd38f6fffe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.393368 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.393388 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm6l7\" (UniqueName: \"kubernetes.io/projected/36f0d01f-16c8-42ce-aa34-e0242d6bc53c-kube-api-access-lm6l7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.393403 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471f3883-ac0a-4781-90e5-77dd38f6fffe-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.398896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.398842 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz" pod "471f3883-ac0a-4781-90e5-77dd38f6fffe" (UID: "471f3883-ac0a-4781-90e5-77dd38f6fffe"). InnerVolumeSpecName "kube-api-access-tgwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:58.427170 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.427143 4918 generic.go:356] "Generic (PLEG): container finished" podID="471f3883-ac0a-4781-90e5-77dd38f6fffe" containerID="a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294" exitCode=0 Oct 30 21:58:58.427303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.427169 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" Oct 30 21:58:58.427303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.427230 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" event={"ID":"471f3883-ac0a-4781-90e5-77dd38f6fffe","Type":"ContainerDied","Data":"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294"} Oct 30 21:58:58.427303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.427274 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf" event={"ID":"471f3883-ac0a-4781-90e5-77dd38f6fffe","Type":"ContainerDied","Data":"1f27e1ab45b662f801e92e4ab2b23519a84b40bf2ea3f2a3851ef1171feabc9f"} Oct 30 21:58:58.427303 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.427301 4918 scope.go:122] "RemoveContainer" containerID="a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294" Oct 30 21:58:58.429079 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.429057 4918 generic.go:356] "Generic (PLEG): container finished" podID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" containerID="ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4" exitCode=0 Oct 30 21:58:58.429262 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.429122 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" Oct 30 21:58:58.429349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.429140 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" event={"ID":"36f0d01f-16c8-42ce-aa34-e0242d6bc53c","Type":"ContainerDied","Data":"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4"} Oct 30 21:58:58.429412 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.429388 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw" event={"ID":"36f0d01f-16c8-42ce-aa34-e0242d6bc53c","Type":"ContainerDied","Data":"129d4f6d3a18d4a574d08a0533638942d74bf6e3302044553603f3115fd00720"} Oct 30 21:58:58.441664 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.441643 4918 scope.go:122] "RemoveContainer" containerID="a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294" Oct 30 21:58:58.442012 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:58.441989 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294\": container with ID starting with a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294 not found: ID does not exist" containerID="a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294" Oct 30 21:58:58.442096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.442023 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294"} err="failed to get container status \"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294\": rpc error: code = NotFound desc = could not find container \"a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294\": container with ID starting with a3c6a37f315a318f939604f1cf14d75cee3ac0e55776ed4fa7c10c3e59435294 not found: ID does not exist" Oct 30 21:58:58.442096 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.442047 4918 scope.go:122] "RemoveContainer" containerID="ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4" Oct 30 21:58:58.454360 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.454338 4918 scope.go:122] "RemoveContainer" containerID="ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4" Oct 30 21:58:58.454656 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:58:58.454624 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4\": container with ID starting with ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4 not found: ID does not exist" containerID="ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4" Oct 30 21:58:58.454754 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.454664 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4"} err="failed to get container status \"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4\": rpc error: code = NotFound desc = could not find container \"ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4\": container with ID starting with ecd7cf1df573506a6f3211183c56c80bd2450264f0abc0469dd94f53e50dc8c4 not found: ID does not exist" Oct 30 21:58:58.494543 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.494515 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwqz\" (UniqueName: \"kubernetes.io/projected/471f3883-ac0a-4781-90e5-77dd38f6fffe-kube-api-access-tgwqz\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.584235 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.584208 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:58:58.590402 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.590382 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-27114d-58b8b5dc56-7tcnw"] Oct 30 21:58:58.741781 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.740774 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:58:58.744756 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:58.744038 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-27114d-6c495c47f-vwmrf"] Oct 30 21:58:59.446332 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:59.446294 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:59.447398 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:59.447371 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" path="/var/lib/kubelet/pods/36f0d01f-16c8-42ce-aa34-e0242d6bc53c/volumes" Oct 30 21:58:59.448194 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:58:59.448168 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471f3883-ac0a-4781-90e5-77dd38f6fffe" path="/var/lib/kubelet/pods/471f3883-ac0a-4781-90e5-77dd38f6fffe/volumes" Oct 30 21:59:03.219001 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:03.218952 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.464186 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:03.464153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:04.545226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:04.545191 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:06.229928 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:06.229891 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:09.270118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:09.270078 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:11.486769 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:11.486734 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:59:11.586730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:11.586688 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:59:11.956440 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:11.956405 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.300895 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:12.300816 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:13.527301 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.527267 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 21:59:13.527990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.527974 4918 cpu_manager.go:414] "containerMap: removing container" podUID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" containerName="payload-processing" Oct 30 21:59:13.527990 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.527990 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" containerName="payload-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.527999 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c28f5ed2-9120-4340-9eb2-b282ee98d733" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528004 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28f5ed2-9120-4340-9eb2-b282ee98d733" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528018 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" containerName="payload-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528023 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" containerName="payload-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528031 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" containerName="payload-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528036 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" containerName="payload-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528056 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528062 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528083 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528088 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528096 4918 cpu_manager.go:414] "containerMap: removing container" podUID="471f3883-ac0a-4781-90e5-77dd38f6fffe" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528101 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471f3883-ac0a-4781-90e5-77dd38f6fffe" containerName="payload-pre-processing" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528114 4918 cpu_manager.go:414] "containerMap: removing container" podUID="42889823-0309-4e14-924a-d0a673eb4ac0" containerName="maas-api" Oct 30 21:59:13.528118 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528119 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42889823-0309-4e14-924a-d0a673eb4ac0" containerName="maas-api" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528227 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00e074a-962f-439a-afc6-8ba3c3abf1d3" containerName="payload-pre-processing" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528241 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e008f318-954f-498c-8fc2-e638bfa5bba9" containerName="main" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528249 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="471f3883-ac0a-4781-90e5-77dd38f6fffe" containerName="payload-pre-processing" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528256 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="42889823-0309-4e14-924a-d0a673eb4ac0" containerName="maas-api" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528265 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f0d01f-16c8-42ce-aa34-e0242d6bc53c" containerName="payload-processing" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528269 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28f5ed2-9120-4340-9eb2-b282ee98d733" containerName="payload-pre-processing" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528277 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bde51c2-d7cb-47c0-be0b-018e87d3449a" containerName="payload-processing" Oct 30 21:59:13.528539 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.528282 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5b3991-546f-4cdd-ad4b-1d80657ab6c5" containerName="payload-processing" Oct 30 21:59:13.532005 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.531989 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.612503 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.612453 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.612669 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.612544 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77j8\" (UniqueName: \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.612792 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.612690 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.612863 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.612806 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.628888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.628860 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bfdf30b9-serving-cert\"" Oct 30 21:59:13.634477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.634453 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 21:59:13.651694 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.651671 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:13.714135 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.714105 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77j8\" (UniqueName: \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.714258 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.714244 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.714357 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.714341 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.714411 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:13.714378 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-serving-cert: secret "maas-api-e2e-mt-bfdf30b9-serving-cert" not found Oct 30 21:59:13.714452 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.714414 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.714497 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:13.714456 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls podName:8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:14.214434721 +0000 UTC m=+2734.907916952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls") pod "maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" (UID: "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590") : secret "maas-api-e2e-mt-bfdf30b9-serving-cert" not found Oct 30 21:59:13.715083 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.715064 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.716850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.716827 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.776564 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.776530 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77j8\" (UniqueName: \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:13.956349 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:13.956313 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:14.220610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:14.220525 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:14.222980 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:14.222963 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") pod \"maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:14.224458 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:14.224432 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:14.446415 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:14.446385 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:14.831279 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:14.831248 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 21:59:14.832029 ip-10-0-132-23 kubenswrapper[4918]: W1030 21:59:14.832005 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc01bf5_b3b5_4336_b6b7_9cfb7fb4a590.slice/crio-155892fca5236c35430219ef7167786217cff07408c136cc603a4e6e9a030342 WatchSource:0}: Error finding container 155892fca5236c35430219ef7167786217cff07408c136cc603a4e6e9a030342: Status 404 returned error can't find the container with id 155892fca5236c35430219ef7167786217cff07408c136cc603a4e6e9a030342 Oct 30 21:59:15.584610 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:15.584572 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" event={"ID":"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590","Type":"ContainerStarted","Data":"155892fca5236c35430219ef7167786217cff07408c136cc603a4e6e9a030342"} Oct 30 21:59:15.812821 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:15.812794 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:59:16.488027 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.487951 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" podUID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" containerName="payload-processing" containerID="cri-o://8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465" gracePeriod=25 Oct 30 21:59:16.587381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.587275 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" podUID="18a26b30-d49d-48b0-a35b-cbf402765197" containerName="payload-pre-processing" containerID="cri-o://8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9" gracePeriod=25 Oct 30 21:59:16.594934 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.594900 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" event={"ID":"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590","Type":"ContainerStarted","Data":"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52"} Oct 30 21:59:16.595839 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.595815 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:16.781600 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.781537 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" podStartSLOduration=2.903040458 podStartE2EDuration="3.781516828s" totalImagesPullingTime="878.47637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:16.763647752 +0000 UTC m=+2737.457129989" watchObservedRunningTime="2026-10-30 21:59:16.781516828 +0000 UTC m=+2737.474999069" Oct 30 21:59:16.826693 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.826666 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:59:16.865751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.864506 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\") pod \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " Oct 30 21:59:16.865751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.864633 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\" (UniqueName: \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\") pod \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\" (UID: \"d9bf3095-687a-48a7-87d1-f9ce40a2ab22\") " Oct 30 21:59:16.866196 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.866164 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume" pod "d9bf3095-687a-48a7-87d1-f9ce40a2ab22" (UID: "d9bf3095-687a-48a7-87d1-f9ce40a2ab22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:16.870719 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.870673 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl" pod "d9bf3095-687a-48a7-87d1-f9ce40a2ab22" (UID: "d9bf3095-687a-48a7-87d1-f9ce40a2ab22"). InnerVolumeSpecName "kube-api-access-p5rfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:16.945532 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.945095 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:59:16.968160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.968115 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.968160 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:16.968155 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rfl\" (UniqueName: \"kubernetes.io/projected/d9bf3095-687a-48a7-87d1-f9ce40a2ab22-kube-api-access-p5rfl\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.068808 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.068734 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\" (UniqueName: \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\") pod \"18a26b30-d49d-48b0-a35b-cbf402765197\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " Oct 30 21:59:17.068956 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.068859 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\") pod \"18a26b30-d49d-48b0-a35b-cbf402765197\" (UID: \"18a26b30-d49d-48b0-a35b-cbf402765197\") " Oct 30 21:59:17.069243 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.069219 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume" pod "18a26b30-d49d-48b0-a35b-cbf402765197" (UID: "18a26b30-d49d-48b0-a35b-cbf402765197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:17.069384 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.069369 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a26b30-d49d-48b0-a35b-cbf402765197-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.070883 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.070859 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn" pod "18a26b30-d49d-48b0-a35b-cbf402765197" (UID: "18a26b30-d49d-48b0-a35b-cbf402765197"). InnerVolumeSpecName "kube-api-access-tpfvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:17.170510 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.170476 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfvn\" (UniqueName: \"kubernetes.io/projected/18a26b30-d49d-48b0-a35b-cbf402765197-kube-api-access-tpfvn\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.605806 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.605769 4918 generic.go:356] "Generic (PLEG): container finished" podID="18a26b30-d49d-48b0-a35b-cbf402765197" containerID="8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9" exitCode=0 Oct 30 21:59:17.606216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.605831 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" Oct 30 21:59:17.606216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.605862 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" event={"ID":"18a26b30-d49d-48b0-a35b-cbf402765197","Type":"ContainerDied","Data":"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9"} Oct 30 21:59:17.606216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.605928 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk" event={"ID":"18a26b30-d49d-48b0-a35b-cbf402765197","Type":"ContainerDied","Data":"820a0c2e59f4d967d8a4ad082ce81ee80b5c81defd470525377fe0b3967d2045"} Oct 30 21:59:17.606216 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.605959 4918 scope.go:122] "RemoveContainer" containerID="8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9" Oct 30 21:59:17.607677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.607650 4918 generic.go:356] "Generic (PLEG): container finished" podID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" containerID="8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465" exitCode=0 Oct 30 21:59:17.607836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.607738 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" Oct 30 21:59:17.607836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.607747 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" event={"ID":"d9bf3095-687a-48a7-87d1-f9ce40a2ab22","Type":"ContainerDied","Data":"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465"} Oct 30 21:59:17.607836 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.607798 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk" event={"ID":"d9bf3095-687a-48a7-87d1-f9ce40a2ab22","Type":"ContainerDied","Data":"a50c2130caf249f29f0547c2a1cc8092092bb929b348952cd2caebfc7c3872c5"} Oct 30 21:59:17.618563 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.618543 4918 scope.go:122] "RemoveContainer" containerID="8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9" Oct 30 21:59:17.618843 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:17.618821 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9\": container with ID starting with 8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9 not found: ID does not exist" containerID="8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9" Oct 30 21:59:17.618933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.618854 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9"} err="failed to get container status \"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9\": rpc error: code = NotFound desc = could not find container \"8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9\": container with ID starting with 8c6bdbe989bc6cf6bcdc73844ce23899871838d2ea3671cd1bd5385c7563aab9 not found: ID does not exist" Oct 30 21:59:17.618933 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.618877 4918 scope.go:122] "RemoveContainer" containerID="8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465" Oct 30 21:59:17.630436 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.630412 4918 scope.go:122] "RemoveContainer" containerID="8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465" Oct 30 21:59:17.630693 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:17.630673 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465\": container with ID starting with 8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465 not found: ID does not exist" containerID="8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465" Oct 30 21:59:17.630796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.630699 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465"} err="failed to get container status \"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465\": rpc error: code = NotFound desc = could not find container \"8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465\": container with ID starting with 8095c592790f703dc941ad17a13b51613065382895299698238e084417a99465 not found: ID does not exist" Oct 30 21:59:17.769153 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.769082 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:59:17.773068 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.773044 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71c0c7-64c7d68f84-ssrgk"] Oct 30 21:59:17.918226 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.918188 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:59:17.926486 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:17.926458 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71c0c7-7d74c9658c-jngwk"] Oct 30 21:59:18.851696 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:18.850598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:19.429424 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:19.429393 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a26b30-d49d-48b0-a35b-cbf402765197" path="/var/lib/kubelet/pods/18a26b30-d49d-48b0-a35b-cbf402765197/volumes" Oct 30 21:59:19.429884 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:19.429867 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" path="/var/lib/kubelet/pods/d9bf3095-687a-48a7-87d1-f9ce40a2ab22/volumes" Oct 30 21:59:22.696370 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:22.696326 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:59:22.696817 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:22.696533 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" podUID="83cb8d26-102d-47e6-aa08-0322310c0884" containerName="maas-api" containerID="cri-o://6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6" gracePeriod=30 Oct 30 21:59:22.844780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:22.844685 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:22.990554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:22.990525 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:59:23.048345 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.048308 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\") pod \"83cb8d26-102d-47e6-aa08-0322310c0884\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " Oct 30 21:59:23.048514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.048369 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") pod \"83cb8d26-102d-47e6-aa08-0322310c0884\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " Oct 30 21:59:23.048514 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.048445 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\" (UniqueName: \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\") pod \"83cb8d26-102d-47e6-aa08-0322310c0884\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " Oct 30 21:59:23.048645 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.048632 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\") pod \"83cb8d26-102d-47e6-aa08-0322310c0884\" (UID: \"83cb8d26-102d-47e6-aa08-0322310c0884\") " Oct 30 21:59:23.048780 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.048689 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca" pod "83cb8d26-102d-47e6-aa08-0322310c0884" (UID: "83cb8d26-102d-47e6-aa08-0322310c0884"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:23.049154 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.049128 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cb8d26-102d-47e6-aa08-0322310c0884-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:23.067927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.050955 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls" pod "83cb8d26-102d-47e6-aa08-0322310c0884" (UID: "83cb8d26-102d-47e6-aa08-0322310c0884"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:23.067927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.052118 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs" pod "83cb8d26-102d-47e6-aa08-0322310c0884" (UID: "83cb8d26-102d-47e6-aa08-0322310c0884"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:23.067927 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.062542 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql" pod "83cb8d26-102d-47e6-aa08-0322310c0884" (UID: "83cb8d26-102d-47e6-aa08-0322310c0884"). InnerVolumeSpecName "kube-api-access-zjbql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:23.150614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.150585 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:23.150614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.150612 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbql\" (UniqueName: \"kubernetes.io/projected/83cb8d26-102d-47e6-aa08-0322310c0884-kube-api-access-zjbql\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:23.150614 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.150623 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cb8d26-102d-47e6-aa08-0322310c0884-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:23.631733 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.629471 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 21:59:23.665475 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.665435 4918 generic.go:356] "Generic (PLEG): container finished" podID="83cb8d26-102d-47e6-aa08-0322310c0884" containerID="6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6" exitCode=0 Oct 30 21:59:23.665755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.665498 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" Oct 30 21:59:23.665755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.665518 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" event={"ID":"83cb8d26-102d-47e6-aa08-0322310c0884","Type":"ContainerDied","Data":"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6"} Oct 30 21:59:23.665755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.665580 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj" event={"ID":"83cb8d26-102d-47e6-aa08-0322310c0884","Type":"ContainerDied","Data":"4ea1d0b74f9456628c0f3ac06f4321ea239dcd8be815f3e33d6c00e380bbde3f"} Oct 30 21:59:23.665755 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.665613 4918 scope.go:122] "RemoveContainer" containerID="6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6" Oct 30 21:59:23.678198 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.678173 4918 scope.go:122] "RemoveContainer" containerID="6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6" Oct 30 21:59:23.678493 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:23.678461 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6\": container with ID starting with 6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6 not found: ID does not exist" containerID="6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6" Oct 30 21:59:23.678601 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.678492 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6"} err="failed to get container status \"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6\": rpc error: code = NotFound desc = could not find container \"6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6\": container with ID starting with 6744834cab8733a6ab7ee2f142deea59d75b98d7ff0577f06f8c85eeb78112e6 not found: ID does not exist" Oct 30 21:59:23.852669 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.852595 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:59:23.862974 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:23.862940 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-27114d-b6667cf58-djkzj"] Oct 30 21:59:24.305480 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:24.305446 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:24.578964 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:24.578894 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:24.798667 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:24.798638 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:25.252146 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:25.252109 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:25.431300 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:25.431268 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cb8d26-102d-47e6-aa08-0322310c0884" path="/var/lib/kubelet/pods/83cb8d26-102d-47e6-aa08-0322310c0884/volumes" Oct 30 21:59:27.949337 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:27.949304 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.142939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:28.142903 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:29.863334 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:29.863288 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:32.560523 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:32.560469 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:32.752977 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:32.752939 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:32.935879 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:32.935848 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:35.216169 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:35.216133 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.291562 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.291530 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:59:39.291998 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.291747 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" podUID="f31eafed-0196-42fd-82ff-8c56442fda11" containerName="maas-api" containerID="cri-o://794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741" gracePeriod=30 Oct 30 21:59:39.438435 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.438393 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.594967 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.594942 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:59:39.673731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.671300 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\") pod \"f31eafed-0196-42fd-82ff-8c56442fda11\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " Oct 30 21:59:39.673731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.671357 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\") pod \"f31eafed-0196-42fd-82ff-8c56442fda11\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " Oct 30 21:59:39.673731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.671476 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") pod \"f31eafed-0196-42fd-82ff-8c56442fda11\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " Oct 30 21:59:39.673731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.671532 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\" (UniqueName: \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\") pod \"f31eafed-0196-42fd-82ff-8c56442fda11\" (UID: \"f31eafed-0196-42fd-82ff-8c56442fda11\") " Oct 30 21:59:39.693730 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.674331 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca" pod "f31eafed-0196-42fd-82ff-8c56442fda11" (UID: "f31eafed-0196-42fd-82ff-8c56442fda11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:39.703554 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.703490 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls" pod "f31eafed-0196-42fd-82ff-8c56442fda11" (UID: "f31eafed-0196-42fd-82ff-8c56442fda11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:39.705244 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.705198 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs" pod "f31eafed-0196-42fd-82ff-8c56442fda11" (UID: "f31eafed-0196-42fd-82ff-8c56442fda11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:39.715947 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.715913 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm" pod "f31eafed-0196-42fd-82ff-8c56442fda11" (UID: "f31eafed-0196-42fd-82ff-8c56442fda11"). InnerVolumeSpecName "kube-api-access-mtmvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:39.773193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.773153 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtmvm\" (UniqueName: \"kubernetes.io/projected/f31eafed-0196-42fd-82ff-8c56442fda11-kube-api-access-mtmvm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.773193 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.773196 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.773386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.773213 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31eafed-0196-42fd-82ff-8c56442fda11-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.773386 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.773227 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31eafed-0196-42fd-82ff-8c56442fda11-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.803987 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.803952 4918 generic.go:356] "Generic (PLEG): container finished" podID="f31eafed-0196-42fd-82ff-8c56442fda11" containerID="794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741" exitCode=0 Oct 30 21:59:39.804150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.803999 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" Oct 30 21:59:39.804150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.804041 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" event={"ID":"f31eafed-0196-42fd-82ff-8c56442fda11","Type":"ContainerDied","Data":"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741"} Oct 30 21:59:39.804150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.804107 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq" event={"ID":"f31eafed-0196-42fd-82ff-8c56442fda11","Type":"ContainerDied","Data":"43eac27baf4805ef19704a9f0977c5cec05a5ef35ac4d0270f1279d6f38f801d"} Oct 30 21:59:39.804150 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.804141 4918 scope.go:122] "RemoveContainer" containerID="794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741" Oct 30 21:59:39.818797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.818414 4918 scope.go:122] "RemoveContainer" containerID="794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741" Oct 30 21:59:39.818896 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:39.818809 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741\": container with ID starting with 794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741 not found: ID does not exist" containerID="794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741" Oct 30 21:59:39.818896 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.818867 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741"} err="failed to get container status \"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741\": rpc error: code = NotFound desc = could not find container \"794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741\": container with ID starting with 794698b0ebf5657ed68fb9c5e1f0b4423ccd94111e79ed18548f8592e4c3c741 not found: ID does not exist" Oct 30 21:59:39.964495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.964463 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:59:39.974090 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:39.974055 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-737a64-845d85c79c-rckbq"] Oct 30 21:59:40.342528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:40.342496 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:59:40.342941 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:40.342689 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" containerID="cri-o://b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa" gracePeriod=30 Oct 30 21:59:40.883926 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:40.883884 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:41.187684 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:41.187584 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:41.408508 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:41.408479 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:41.430344 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:41.430316 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31eafed-0196-42fd-82ff-8c56442fda11" path="/var/lib/kubelet/pods/f31eafed-0196-42fd-82ff-8c56442fda11/volumes" Oct 30 21:59:41.850381 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:41.850350 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:45.227495 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.227462 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:59:45.227887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.227521 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:59:45.338847 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.338811 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:45.540047 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.538325 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:59:45.629650 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.629625 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:59:45.641771 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.641737 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:59:45.785259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785154 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785217 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785259 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785249 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785392 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785453 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785490 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785523 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785567 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785555 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.785950 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.785612 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\") pod \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\" (UID: \"d625a7ac-414e-43ee-a38e-d6a0b65ed6be\") " Oct 30 21:59:45.786939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.786526 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.786939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.786542 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.786939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.786836 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.786939 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.786853 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.787454 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.787430 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:45.789924 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.789889 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:45.790309 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.790285 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "kube-api-access-fj4sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:45.790585 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.790549 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.807858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.807832 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo" pod "d625a7ac-414e-43ee-a38e-d6a0b65ed6be" (UID: "d625a7ac-414e-43ee-a38e-d6a0b65ed6be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:45.807987 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:45.807885 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 21:59:45.859061 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.859028 4918 generic.go:356] "Generic (PLEG): container finished" podID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerID="b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa" exitCode=0 Oct 30 21:59:45.859289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.859085 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" Oct 30 21:59:45.859289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.859235 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerDied","Data":"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa"} Oct 30 21:59:45.859289 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.859281 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b" event={"ID":"d625a7ac-414e-43ee-a38e-d6a0b65ed6be","Type":"ContainerDied","Data":"06424e29c1b2f04ecc3352e1ffdab98d836599a43da993b7b84aaf1411aba900"} Oct 30 21:59:45.859528 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.859312 4918 scope.go:122] "RemoveContainer" containerID="b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa" Oct 30 21:59:45.881866 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.881834 4918 scope.go:122] "RemoveContainer" containerID="03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75" Oct 30 21:59:45.887213 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887185 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887221 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887240 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887256 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887273 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj4sp\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-kube-api-access-fj4sp\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887291 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887306 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887319 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887320 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.887627 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.887336 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d625a7ac-414e-43ee-a38e-d6a0b65ed6be-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.904016 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.903938 4918 scope.go:122] "RemoveContainer" containerID="b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa" Oct 30 21:59:45.904580 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:45.904531 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa\": container with ID starting with b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa not found: ID does not exist" containerID="b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa" Oct 30 21:59:45.904765 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.904726 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa"} err="failed to get container status \"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa\": rpc error: code = NotFound desc = could not find container \"b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa\": container with ID starting with b8d262f477fb469defca8062f553d75c5b7778f86a2228ac82ee8853c59839fa not found: ID does not exist" Oct 30 21:59:45.904882 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.904780 4918 scope.go:122] "RemoveContainer" containerID="03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75" Oct 30 21:59:45.905204 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:45.905180 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75\": container with ID starting with 03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75 not found: ID does not exist" containerID="03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75" Oct 30 21:59:45.905292 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.905211 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75"} err="failed to get container status \"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75\": rpc error: code = NotFound desc = could not find container \"03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75\": container with ID starting with 03242ffbc43910e55e48de7428f7528aab379a3513ffb66f48cde7f0c64aab75 not found: ID does not exist" Oct 30 21:59:45.942421 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.942393 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:45.985509 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.985481 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:59:45.989611 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:45.989586 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-kq56b"] Oct 30 21:59:46.169797 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:46.169755 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:47.431717 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:47.431682 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" path="/var/lib/kubelet/pods/d625a7ac-414e-43ee-a38e-d6a0b65ed6be/volumes" Oct 30 21:59:48.844376 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:48.844343 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:49.050065 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:49.050032 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:49.256432 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:49.256395 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:50.540076 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.540012 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" podUID="d247f88e-0b87-49be-a775-3454fbf362f6" containerName="payload-processing" containerID="cri-o://1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c" gracePeriod=25 Oct 30 21:59:50.642227 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.642194 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" podUID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" containerName="payload-pre-processing" containerID="cri-o://e8abb7810dec676bf8f12025b37521892f2d337adfa1b10afd4b0f982c69bc5c" gracePeriod=25 Oct 30 21:59:50.849723 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.849580 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:59:50.910044 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.910014 4918 generic.go:356] "Generic (PLEG): container finished" podID="d247f88e-0b87-49be-a775-3454fbf362f6" containerID="1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c" exitCode=0 Oct 30 21:59:50.910365 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.910223 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" event={"ID":"d247f88e-0b87-49be-a775-3454fbf362f6","Type":"ContainerDied","Data":"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c"} Oct 30 21:59:50.910477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.910391 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" event={"ID":"d247f88e-0b87-49be-a775-3454fbf362f6","Type":"ContainerDied","Data":"04a86b4c5e4384d04901f1f7195101636700ccc3aea938ba35968425996e9e1e"} Oct 30 21:59:50.910477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.910418 4918 scope.go:122] "RemoveContainer" containerID="1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c" Oct 30 21:59:50.910477 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.910254 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj" Oct 30 21:59:50.912744 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.912688 4918 generic.go:356] "Generic (PLEG): container finished" podID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" containerID="e8abb7810dec676bf8f12025b37521892f2d337adfa1b10afd4b0f982c69bc5c" exitCode=0 Oct 30 21:59:50.912854 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.912788 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" event={"ID":"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576","Type":"ContainerDied","Data":"e8abb7810dec676bf8f12025b37521892f2d337adfa1b10afd4b0f982c69bc5c"} Oct 30 21:59:50.922850 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.922834 4918 scope.go:122] "RemoveContainer" containerID="1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c" Oct 30 21:59:50.923147 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:50.923128 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c\": container with ID starting with 1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c not found: ID does not exist" containerID="1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c" Oct 30 21:59:50.923206 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.923154 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c"} err="failed to get container status \"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c\": rpc error: code = NotFound desc = could not find container \"1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c\": container with ID starting with 1a9c208c3c6df48c1a0516d88052615d086c9dac37054c0f24765f3a11c5fe2c not found: ID does not exist" Oct 30 21:59:50.923399 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.923378 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:59:50.951182 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.951158 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\" (UniqueName: \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\") pod \"d247f88e-0b87-49be-a775-3454fbf362f6\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " Oct 30 21:59:50.951372 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.951350 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\") pod \"d247f88e-0b87-49be-a775-3454fbf362f6\" (UID: \"d247f88e-0b87-49be-a775-3454fbf362f6\") " Oct 30 21:59:50.952247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.951780 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume" pod "d247f88e-0b87-49be-a775-3454fbf362f6" (UID: "d247f88e-0b87-49be-a775-3454fbf362f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:50.953551 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:50.953527 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw" pod "d247f88e-0b87-49be-a775-3454fbf362f6" (UID: "d247f88e-0b87-49be-a775-3454fbf362f6"). InnerVolumeSpecName "kube-api-access-tvvlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:51.052751 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.052694 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\") pod \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " Oct 30 21:59:51.052887 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.052854 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\" (UniqueName: \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\") pod \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\" (UID: \"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576\") " Oct 30 21:59:51.053105 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.053079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume" pod "981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" (UID: "981fcd7c-0a93-4acb-aa01-2d0cf6eb4576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:51.053270 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.053255 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:51.053307 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.053274 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvvlw\" (UniqueName: \"kubernetes.io/projected/d247f88e-0b87-49be-a775-3454fbf362f6-kube-api-access-tvvlw\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:51.053307 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.053284 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d247f88e-0b87-49be-a775-3454fbf362f6-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:51.054949 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.054931 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9" pod "981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" (UID: "981fcd7c-0a93-4acb-aa01-2d0cf6eb4576"). InnerVolumeSpecName "kube-api-access-pgzh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:51.154281 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.154249 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzh9\" (UniqueName: \"kubernetes.io/projected/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576-kube-api-access-pgzh9\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:51.244654 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.244598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:51.338823 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.338783 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:59:51.342716 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.342676 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ed6135-74cd869c8d-28gsj"] Oct 30 21:59:51.429679 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.429583 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d247f88e-0b87-49be-a775-3454fbf362f6" path="/var/lib/kubelet/pods/d247f88e-0b87-49be-a775-3454fbf362f6/volumes" Oct 30 21:59:51.922782 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.922753 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" event={"ID":"981fcd7c-0a93-4acb-aa01-2d0cf6eb4576","Type":"ContainerDied","Data":"37ff4ae74d4d820e39cfb70c72d92411ffa079dad0991ded037a4d679039dd71"} Oct 30 21:59:51.923247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.922807 4918 scope.go:122] "RemoveContainer" containerID="e8abb7810dec676bf8f12025b37521892f2d337adfa1b10afd4b0f982c69bc5c" Oct 30 21:59:51.923247 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:51.922767 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv" Oct 30 21:59:52.079858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:52.079821 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:59:52.126396 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:52.126366 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ed6135-56f649c84c-8cwbv"] Oct 30 21:59:53.429016 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:53.428974 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" path="/var/lib/kubelet/pods/981fcd7c-0a93-4acb-aa01-2d0cf6eb4576/volumes" Oct 30 21:59:55.676731 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.676670 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:59:55.677275 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.676909 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" podUID="f2607bb4-a4b4-4809-9413-3fe1520516ea" containerName="maas-api" containerID="cri-o://9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27" gracePeriod=30 Oct 30 21:59:55.862971 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.862934 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:55.971903 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.971875 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:59:55.972416 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.972392 4918 generic.go:356] "Generic (PLEG): container finished" podID="f2607bb4-a4b4-4809-9413-3fe1520516ea" containerID="9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27" exitCode=0 Oct 30 21:59:55.972521 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.972466 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" event={"ID":"f2607bb4-a4b4-4809-9413-3fe1520516ea","Type":"ContainerDied","Data":"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27"} Oct 30 21:59:55.972561 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.972529 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" event={"ID":"f2607bb4-a4b4-4809-9413-3fe1520516ea","Type":"ContainerDied","Data":"d848d659345219ff47b37537a55f8333b76145e942d3d4161e5fd46488cd7375"} Oct 30 21:59:55.972612 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.972563 4918 scope.go:122] "RemoveContainer" containerID="9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27" Oct 30 21:59:55.986542 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.986521 4918 scope.go:122] "RemoveContainer" containerID="9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27" Oct 30 21:59:55.986882 ip-10-0-132-23 kubenswrapper[4918]: E1030 21:59:55.986861 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27\": container with ID starting with 9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27 not found: ID does not exist" containerID="9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27" Oct 30 21:59:55.986969 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:55.986894 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27"} err="failed to get container status \"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27\": rpc error: code = NotFound desc = could not find container \"9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27\": container with ID starting with 9e0710675b80c55946c3edc1e3df15420ad47531f1c3269a01c2f2acc6a89d27 not found: ID does not exist" Oct 30 21:59:56.010556 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.010533 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\" (UniqueName: \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\") pod \"f2607bb4-a4b4-4809-9413-3fe1520516ea\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " Oct 30 21:59:56.010677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.010591 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\") pod \"f2607bb4-a4b4-4809-9413-3fe1520516ea\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " Oct 30 21:59:56.010677 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.010652 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") pod \"f2607bb4-a4b4-4809-9413-3fe1520516ea\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " Oct 30 21:59:56.010848 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.010830 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\") pod \"f2607bb4-a4b4-4809-9413-3fe1520516ea\" (UID: \"f2607bb4-a4b4-4809-9413-3fe1520516ea\") " Oct 30 21:59:56.011310 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.011279 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca" pod "f2607bb4-a4b4-4809-9413-3fe1520516ea" (UID: "f2607bb4-a4b4-4809-9413-3fe1520516ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:56.015743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.012870 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls" pod "f2607bb4-a4b4-4809-9413-3fe1520516ea" (UID: "f2607bb4-a4b4-4809-9413-3fe1520516ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:56.015743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.012931 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr" pod "f2607bb4-a4b4-4809-9413-3fe1520516ea" (UID: "f2607bb4-a4b4-4809-9413-3fe1520516ea"). InnerVolumeSpecName "kube-api-access-s8jsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:56.015743 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.012954 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs" pod "f2607bb4-a4b4-4809-9413-3fe1520516ea" (UID: "f2607bb4-a4b4-4809-9413-3fe1520516ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:56.112695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.112651 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jsr\" (UniqueName: \"kubernetes.io/projected/f2607bb4-a4b4-4809-9413-3fe1520516ea-kube-api-access-s8jsr\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.112695 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.112695 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.112992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.112726 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2607bb4-a4b4-4809-9413-3fe1520516ea-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.112992 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.112739 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2607bb4-a4b4-4809-9413-3fe1520516ea-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.981888 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:56.981849 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj" Oct 30 21:59:57.152858 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:57.152828 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:59:57.196796 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:57.196763 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-167561-6d689d959f-dgsmj"] Oct 30 21:59:57.429489 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:57.429457 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2607bb4-a4b4-4809-9413-3fe1520516ea" path="/var/lib/kubelet/pods/f2607bb4-a4b4-4809-9413-3fe1520516ea/volumes" Oct 30 21:59:57.795202 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:57.795121 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.453860 ip-10-0-132-23 kubenswrapper[4918]: I1030 21:59:58.453824 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:01.677100 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:01.677058 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 22:00:01.782092 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:01.782062 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 22:00:02.139228 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:02.139188 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:02.400223 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:02.400137 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:03.021883 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:03.021842 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:04.962557 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:04.962524 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:05.234247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.234173 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:05.573952 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.573869 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:00:05.574560 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574542 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" containerName="payload-processing" Oct 30 22:00:05.574560 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574560 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" containerName="payload-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574573 4918 cpu_manager.go:414] "containerMap: removing container" podUID="83cb8d26-102d-47e6-aa08-0322310c0884" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574579 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cb8d26-102d-47e6-aa08-0322310c0884" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574586 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f31eafed-0196-42fd-82ff-8c56442fda11" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574590 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31eafed-0196-42fd-82ff-8c56442fda11" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574618 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d247f88e-0b87-49be-a775-3454fbf362f6" containerName="payload-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574623 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d247f88e-0b87-49be-a775-3454fbf362f6" containerName="payload-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574631 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f2607bb4-a4b4-4809-9413-3fe1520516ea" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574636 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2607bb4-a4b4-4809-9413-3fe1520516ea" containerName="maas-api" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574643 4918 cpu_manager.go:414] "containerMap: removing container" podUID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" containerName="payload-pre-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574648 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" containerName="payload-pre-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574658 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574662 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574679 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574683 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574697 4918 cpu_manager.go:414] "containerMap: removing container" podUID="18a26b30-d49d-48b0-a35b-cbf402765197" containerName="payload-pre-processing" Oct 30 22:00:05.574762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574702 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a26b30-d49d-48b0-a35b-cbf402765197" containerName="payload-pre-processing" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574816 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574833 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a26b30-d49d-48b0-a35b-cbf402765197" containerName="payload-pre-processing" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574844 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bf3095-687a-48a7-87d1-f9ce40a2ab22" containerName="payload-processing" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574850 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cb8d26-102d-47e6-aa08-0322310c0884" containerName="maas-api" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574856 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31eafed-0196-42fd-82ff-8c56442fda11" containerName="maas-api" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574867 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d247f88e-0b87-49be-a775-3454fbf362f6" containerName="payload-processing" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574873 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2607bb4-a4b4-4809-9413-3fe1520516ea" containerName="maas-api" Oct 30 22:00:05.575605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.574883 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="981fcd7c-0a93-4acb-aa01-2d0cf6eb4576" containerName="payload-pre-processing" Oct 30 22:00:05.580266 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.580243 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.640932 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.640898 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:00:05.649934 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.649898 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650128 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.649939 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650128 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650057 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650253 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650160 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650291 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650370 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650317 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650370 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650340 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwp8\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650481 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.650540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.650485 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.694078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.694037 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bfdf30b9-openshift-default-dockercfg-pzmxv\"" Oct 30 22:00:05.751497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751454 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.751497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751495 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.751892 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751628 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.751892 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751687 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.751892 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751870 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752207 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.751904 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752291 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752229 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwp8\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752291 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752271 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752308 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752331 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752531 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752503 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752523 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752697 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752675 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.752818 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.752805 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.754152 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.754122 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.754354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.754337 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.810379 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.810317 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.812545 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.812521 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwp8\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:05.894516 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:05.894487 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:06.338646 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:00:06.338589 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06501d6_4873_4e09_a1c5_0620ba89da6d.slice/crio-0771bccfc1789bf3f5622fdd7fe300d0173e44c865b2738bf8065545176f13c6 WatchSource:0}: Error finding container 0771bccfc1789bf3f5622fdd7fe300d0173e44c865b2738bf8065545176f13c6: Status 404 returned error can't find the container with id 0771bccfc1789bf3f5622fdd7fe300d0173e44c865b2738bf8065545176f13c6 Oct 30 22:00:06.342814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.342785 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:00:06.343749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.343685 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:06.343932 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.343912 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:06.344056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.344040 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:06.678276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.678230 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" podUID="bacc135d-2181-4a53-a5b9-791939b29f72" containerName="payload-processing" containerID="cri-o://cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a" gracePeriod=25 Oct 30 22:00:06.782837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:06.782771 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" podUID="0d1a71dc-d561-46bc-9ddb-c2607217d015" containerName="payload-pre-processing" containerID="cri-o://19bc4d5fb9d94b5c5e002b5a1559205aff66441b0ba4bc3a13a170673f6f9768" gracePeriod=25 Oct 30 22:00:07.005734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.001963 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 22:00:07.065140 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.064373 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" event={"ID":"c06501d6-4873-4e09-a1c5-0620ba89da6d","Type":"ContainerStarted","Data":"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6"} Oct 30 22:00:07.065140 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.064431 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" event={"ID":"c06501d6-4873-4e09-a1c5-0620ba89da6d","Type":"ContainerStarted","Data":"0771bccfc1789bf3f5622fdd7fe300d0173e44c865b2738bf8065545176f13c6"} Oct 30 22:00:07.068106 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.067960 4918 generic.go:356] "Generic (PLEG): container finished" podID="0d1a71dc-d561-46bc-9ddb-c2607217d015" containerID="19bc4d5fb9d94b5c5e002b5a1559205aff66441b0ba4bc3a13a170673f6f9768" exitCode=0 Oct 30 22:00:07.068106 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.068049 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" event={"ID":"0d1a71dc-d561-46bc-9ddb-c2607217d015","Type":"ContainerDied","Data":"19bc4d5fb9d94b5c5e002b5a1559205aff66441b0ba4bc3a13a170673f6f9768"} Oct 30 22:00:07.077744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.069773 4918 generic.go:356] "Generic (PLEG): container finished" podID="bacc135d-2181-4a53-a5b9-791939b29f72" containerID="cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a" exitCode=0 Oct 30 22:00:07.077744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.069943 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" event={"ID":"bacc135d-2181-4a53-a5b9-791939b29f72","Type":"ContainerDied","Data":"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a"} Oct 30 22:00:07.077744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.069984 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" event={"ID":"bacc135d-2181-4a53-a5b9-791939b29f72","Type":"ContainerDied","Data":"4856df323ad66f3f14370697e660543f3f733b39309359c8e704c7f0412f7d0a"} Oct 30 22:00:07.077744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.070011 4918 scope.go:122] "RemoveContainer" containerID="cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a" Oct 30 22:00:07.077744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.070165 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl" Oct 30 22:00:07.103461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.103426 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\") pod \"bacc135d-2181-4a53-a5b9-791939b29f72\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " Oct 30 22:00:07.104354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.104184 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume" pod "bacc135d-2181-4a53-a5b9-791939b29f72" (UID: "bacc135d-2181-4a53-a5b9-791939b29f72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:07.104354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.104328 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\" (UniqueName: \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\") pod \"bacc135d-2181-4a53-a5b9-791939b29f72\" (UID: \"bacc135d-2181-4a53-a5b9-791939b29f72\") " Oct 30 22:00:07.105268 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.105241 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacc135d-2181-4a53-a5b9-791939b29f72-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.113654 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.111378 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs" pod "bacc135d-2181-4a53-a5b9-791939b29f72" (UID: "bacc135d-2181-4a53-a5b9-791939b29f72"). InnerVolumeSpecName "kube-api-access-zpxvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:07.117876 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.117842 4918 scope.go:122] "RemoveContainer" containerID="cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a" Oct 30 22:00:07.119822 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:07.119788 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a\": container with ID starting with cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a not found: ID does not exist" containerID="cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a" Oct 30 22:00:07.119994 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.119965 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a"} err="failed to get container status \"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a\": rpc error: code = NotFound desc = could not find container \"cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a\": container with ID starting with cd0a0134da2ffad358d514bd4fdb748d1c6070dfd1f931662deb2cdd3106c81a not found: ID does not exist" Oct 30 22:00:07.142854 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.142830 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 22:00:07.206517 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.206432 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpxvs\" (UniqueName: \"kubernetes.io/projected/bacc135d-2181-4a53-a5b9-791939b29f72-kube-api-access-zpxvs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.266860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.266803 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podStartSLOduration=2.266789187 podStartE2EDuration="2.266789187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.262319816 +0000 UTC m=+2787.955802052" watchObservedRunningTime="2026-10-30 22:00:07.266789187 +0000 UTC m=+2787.960271423" Oct 30 22:00:07.307379 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.307344 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\" (UniqueName: \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\") pod \"0d1a71dc-d561-46bc-9ddb-c2607217d015\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " Oct 30 22:00:07.307554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.307498 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\") pod \"0d1a71dc-d561-46bc-9ddb-c2607217d015\" (UID: \"0d1a71dc-d561-46bc-9ddb-c2607217d015\") " Oct 30 22:00:07.307902 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.307878 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume" pod "0d1a71dc-d561-46bc-9ddb-c2607217d015" (UID: "0d1a71dc-d561-46bc-9ddb-c2607217d015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:07.308019 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.308006 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1a71dc-d561-46bc-9ddb-c2607217d015-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.309499 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.309475 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4" pod "0d1a71dc-d561-46bc-9ddb-c2607217d015" (UID: "0d1a71dc-d561-46bc-9ddb-c2607217d015"). InnerVolumeSpecName "kube-api-access-7s9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:07.408981 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.408951 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9r4\" (UniqueName: \"kubernetes.io/projected/0d1a71dc-d561-46bc-9ddb-c2607217d015-kube-api-access-7s9r4\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.417476 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.417447 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:07.600566 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.600466 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 22:00:07.604082 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.604050 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2df7b6-79bc769774-dr9xl"] Oct 30 22:00:07.608966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.608933 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:07.894933 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.894887 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:07.897119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.897088 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:00:07.897235 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:07.897146 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:00:08.101310 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.101275 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" Oct 30 22:00:08.101483 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.101274 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g" event={"ID":"0d1a71dc-d561-46bc-9ddb-c2607217d015","Type":"ContainerDied","Data":"56b7fa68e00e8666d30a6c8e84c6a0fe90c25b5604294508708446442e7be8b8"} Oct 30 22:00:08.101483 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.101449 4918 scope.go:122] "RemoveContainer" containerID="19bc4d5fb9d94b5c5e002b5a1559205aff66441b0ba4bc3a13a170673f6f9768" Oct 30 22:00:08.166054 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.162948 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.278739 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.278058 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 22:00:08.295743 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.286202 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2df7b6-8648547c67-5wl6g"] Oct 30 22:00:08.385731 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.382645 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.675212 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.675134 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.895648 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.895065 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:00:08.895648 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:08.895165 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:00:09.429264 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:09.429227 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1a71dc-d561-46bc-9ddb-c2607217d015" path="/var/lib/kubelet/pods/0d1a71dc-d561-46bc-9ddb-c2607217d015/volumes" Oct 30 22:00:09.429688 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:09.429674 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacc135d-2181-4a53-a5b9-791939b29f72" path="/var/lib/kubelet/pods/bacc135d-2181-4a53-a5b9-791939b29f72/volumes" Oct 30 22:00:09.538055 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:09.538014 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.898282 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:09.898250 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:00:09.898886 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:09.898853 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:00:10.901896 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:10.901852 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:00:10.902566 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:10.902532 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:00:10.910422 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:10.910402 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:11.294262 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:11.294170 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:11.895064 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:11.895028 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:00:11.895346 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:11.895311 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 22:00:12.900269 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:12.900236 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:12.900785 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:12.900767 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:12.901496 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:12.901475 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:00:13.035016 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:13.034979 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.023295 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:15.023262 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.319277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:15.319186 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.806831 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:15.806795 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:00:17.108629 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:17.108586 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:18.148732 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:18.148668 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:18.459270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:18.459185 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:20.667964 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.667929 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:00:20.668579 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668565 4918 cpu_manager.go:414] "containerMap: removing container" podUID="bacc135d-2181-4a53-a5b9-791939b29f72" containerName="payload-processing" Oct 30 22:00:20.668637 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668579 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacc135d-2181-4a53-a5b9-791939b29f72" containerName="payload-processing" Oct 30 22:00:20.668637 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668608 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0d1a71dc-d561-46bc-9ddb-c2607217d015" containerName="payload-pre-processing" Oct 30 22:00:20.668637 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668616 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1a71dc-d561-46bc-9ddb-c2607217d015" containerName="payload-pre-processing" Oct 30 22:00:20.668784 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668772 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d625a7ac-414e-43ee-a38e-d6a0b65ed6be" containerName="istio-proxy" Oct 30 22:00:20.668830 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668788 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacc135d-2181-4a53-a5b9-791939b29f72" containerName="payload-processing" Oct 30 22:00:20.668830 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.668798 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1a71dc-d561-46bc-9ddb-c2607217d015" containerName="payload-pre-processing" Oct 30 22:00:20.673934 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.673912 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.778415 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.778378 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:00:20.839571 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839542 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.839784 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839589 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.839784 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839696 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.839914 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839794 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.839914 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839836 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.839993 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.839920 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.840174 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.840149 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.840237 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.840217 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzb4f\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.840290 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.840273 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941243 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941150 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941243 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941203 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941243 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941234 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941282 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941318 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941363 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941479 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941739 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941538 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzb4f\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.941739 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.941595 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.942172 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.942142 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.942400 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.942378 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.942916 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.942893 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.944209 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.943934 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.944786 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.944758 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.946323 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.946299 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:20.947309 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:20.947285 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:21.000396 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.000363 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:21.002467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.002436 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzb4f\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:21.291299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.291215 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:21.736870 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.736833 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:00:21.755825 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:00:21.755788 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf60cf13_cbc8_4742_b7d9_c0b9e8f38aec.slice/crio-f15f14946f0f11574d7e479a5c382f3156d66b08f535f8b6cd9fdbd14dc88fb8 WatchSource:0}: Error finding container f15f14946f0f11574d7e479a5c382f3156d66b08f535f8b6cd9fdbd14dc88fb8: Status 404 returned error can't find the container with id f15f14946f0f11574d7e479a5c382f3156d66b08f535f8b6cd9fdbd14dc88fb8 Oct 30 22:00:21.758396 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.758362 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:21.758467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.758428 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:21.758467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:21.758461 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:22.227149 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.227108 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" event={"ID":"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec","Type":"ContainerStarted","Data":"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214"} Oct 30 22:00:22.227360 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.227169 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" event={"ID":"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec","Type":"ContainerStarted","Data":"f15f14946f0f11574d7e479a5c382f3156d66b08f535f8b6cd9fdbd14dc88fb8"} Oct 30 22:00:22.291755 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.291696 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:22.293337 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.293304 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:22.293478 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.293377 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:22.457678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:22.457621 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podStartSLOduration=2.457605699 podStartE2EDuration="2.457605699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:22.411351405 +0000 UTC m=+2803.104833642" watchObservedRunningTime="2026-10-30 22:00:22.457605699 +0000 UTC m=+2803.151087934" Oct 30 22:00:23.237132 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:23.237098 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.296729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:23.294376 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:23.296729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:23.294436 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:24.295728 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:24.295338 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:24.295728 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:24.295399 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:25.293871 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:25.293827 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.294159 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:25.294127 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:26.292187 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:26.292156 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.292817 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:26.292767 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:27.294411 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:27.294374 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.295066 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:27.295018 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 22:00:28.406808 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:28.406772 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:28.816770 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:28.816644 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:28.817238 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:28.817212 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:28.817985 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:28.817963 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:00:32.593241 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:32.593209 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 22:00:32.593662 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:32.593405 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" podUID="db507623-9d17-4e5e-b1b9-d1d9084b106b" containerName="maas-api" containerID="cri-o://3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553" gracePeriod=30 Oct 30 22:00:32.807212 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:32.807141 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:32.911195 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:32.911162 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 22:00:33.036218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.036179 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\") pod \"db507623-9d17-4e5e-b1b9-d1d9084b106b\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " Oct 30 22:00:33.036426 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.036384 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\" (UniqueName: \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\") pod \"db507623-9d17-4e5e-b1b9-d1d9084b106b\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " Oct 30 22:00:33.036494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.036430 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\") pod \"db507623-9d17-4e5e-b1b9-d1d9084b106b\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " Oct 30 22:00:33.036558 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.036526 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") pod \"db507623-9d17-4e5e-b1b9-d1d9084b106b\" (UID: \"db507623-9d17-4e5e-b1b9-d1d9084b106b\") " Oct 30 22:00:33.037445 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.037410 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca" pod "db507623-9d17-4e5e-b1b9-d1d9084b106b" (UID: "db507623-9d17-4e5e-b1b9-d1d9084b106b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:33.041156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.041114 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5" pod "db507623-9d17-4e5e-b1b9-d1d9084b106b" (UID: "db507623-9d17-4e5e-b1b9-d1d9084b106b"). InnerVolumeSpecName "kube-api-access-66xb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:33.042794 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.042745 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls" pod "db507623-9d17-4e5e-b1b9-d1d9084b106b" (UID: "db507623-9d17-4e5e-b1b9-d1d9084b106b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:33.043736 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.043682 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs" pod "db507623-9d17-4e5e-b1b9-d1d9084b106b" (UID: "db507623-9d17-4e5e-b1b9-d1d9084b106b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:33.142838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.142734 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:33.143841 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.143812 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xb5\" (UniqueName: \"kubernetes.io/projected/db507623-9d17-4e5e-b1b9-d1d9084b106b-kube-api-access-66xb5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:33.144041 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.144024 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db507623-9d17-4e5e-b1b9-d1d9084b106b-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:33.144155 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.144139 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db507623-9d17-4e5e-b1b9-d1d9084b106b-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:33.329306 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.329265 4918 generic.go:356] "Generic (PLEG): container finished" podID="db507623-9d17-4e5e-b1b9-d1d9084b106b" containerID="3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553" exitCode=0 Oct 30 22:00:33.329498 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.329309 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" Oct 30 22:00:33.329498 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.329325 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" event={"ID":"db507623-9d17-4e5e-b1b9-d1d9084b106b","Type":"ContainerDied","Data":"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553"} Oct 30 22:00:33.329498 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.329410 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf" event={"ID":"db507623-9d17-4e5e-b1b9-d1d9084b106b","Type":"ContainerDied","Data":"7f2f819422936bf6115efbd9c85d4e57a78eb92182e52bf1c238cadab9634b7e"} Oct 30 22:00:33.329498 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.329444 4918 scope.go:122] "RemoveContainer" containerID="3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553" Oct 30 22:00:33.341394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.341377 4918 scope.go:122] "RemoveContainer" containerID="3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553" Oct 30 22:00:33.341665 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:33.341647 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553\": container with ID starting with 3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553 not found: ID does not exist" containerID="3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553" Oct 30 22:00:33.341740 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.341675 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553"} err="failed to get container status \"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553\": rpc error: code = NotFound desc = could not find container \"3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553\": container with ID starting with 3d1ee82f9b0a8cb6a25a761e76f30f7de24c276b5c47a9606b557f73f6dc0553 not found: ID does not exist" Oct 30 22:00:33.487585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.487518 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 22:00:33.599960 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:33.599923 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-554f46-867d78d79-98njf"] Oct 30 22:00:34.032717 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:34.032670 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:34.262224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:34.262193 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:34.487724 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:34.487683 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:34.725565 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:34.725530 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.269042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.269007 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.429494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.429460 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db507623-9d17-4e5e-b1b9-d1d9084b106b" path="/var/lib/kubelet/pods/db507623-9d17-4e5e-b1b9-d1d9084b106b/volumes" Oct 30 22:00:35.511247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.511215 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.825059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.825027 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:00:35.826588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.825918 4918 cpu_manager.go:414] "containerMap: removing container" podUID="db507623-9d17-4e5e-b1b9-d1d9084b106b" containerName="maas-api" Oct 30 22:00:35.826588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.825940 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db507623-9d17-4e5e-b1b9-d1d9084b106b" containerName="maas-api" Oct 30 22:00:35.826588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.826138 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="db507623-9d17-4e5e-b1b9-d1d9084b106b" containerName="maas-api" Oct 30 22:00:35.831548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.831528 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.876322 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.876291 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:00:35.902596 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.902564 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.912525 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912495 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.912753 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912691 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.912848 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912787 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.912848 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912820 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.912956 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912849 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.912956 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.912928 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wdg\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.913069 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.913033 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.913133 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.913105 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:35.913190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:35.913140 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014377 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014338 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014442 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014467 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014487 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014505 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014522 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wdg\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014559 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014618 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014653 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014814 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.014861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014831 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.015126 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.014892 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.015181 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.015138 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.015293 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.015271 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.017049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.017026 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.017202 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.017166 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.072852 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.072830 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.072995 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.072959 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wdg\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.146080 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.146053 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:36.167179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.167150 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.456453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.456380 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.585516 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.585488 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:00:36.587113 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:00:36.587081 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8244505_8b0c_4f00_aeca_abf790ed1d7d.slice/crio-59c5c2c66eee119562d2d052670bad507b1c1d75de30feb0d173b8df9747434b WatchSource:0}: Error finding container 59c5c2c66eee119562d2d052670bad507b1c1d75de30feb0d173b8df9747434b: Status 404 returned error can't find the container with id 59c5c2c66eee119562d2d052670bad507b1c1d75de30feb0d173b8df9747434b Oct 30 22:00:36.590072 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.590036 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:36.590169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.590108 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:36.590169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:36.590137 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:37.373730 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:37.369916 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" event={"ID":"b8244505-8b0c-4f00-aeca-abf790ed1d7d","Type":"ContainerStarted","Data":"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0"} Oct 30 22:00:37.373730 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:37.369983 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" event={"ID":"b8244505-8b0c-4f00-aeca-abf790ed1d7d","Type":"ContainerStarted","Data":"59c5c2c66eee119562d2d052670bad507b1c1d75de30feb0d173b8df9747434b"} Oct 30 22:00:37.539635 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:37.539572 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podStartSLOduration=2.5395584490000003 podStartE2EDuration="2.539558449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:37.53637841 +0000 UTC m=+2818.229860647" watchObservedRunningTime="2026-10-30 22:00:37.539558449 +0000 UTC m=+2818.233040685" Oct 30 22:00:38.146572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:38.146534 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:38.148276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:38.148245 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:00:38.148423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:38.148305 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:00:39.146658 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:39.146624 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:00:39.147170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:39.146723 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:00:40.147056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:40.147021 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:00:40.147466 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:40.147071 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:00:40.928852 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:40.928793 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:41.147397 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:41.147355 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.147928 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:41.147415 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:00:42.146626 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:42.146591 4918 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.146820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:42.146645 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 22:00:43.151695 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.151659 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:43.152230 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.152207 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:43.153205 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.153178 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:00:43.168536 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.168469 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:43.514530 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.514452 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:43.815373 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:43.815294 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:44.622429 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:44.622391 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:44.899121 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:44.899020 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:45.139841 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:45.139810 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:45.467600 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:45.467541 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:45.802846 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:45.802767 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:00:45.853786 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:45.853657 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:46.070123 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:46.070048 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:50.395648 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:50.395603 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 22:00:50.419371 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:50.419340 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:50.668638 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:50.668554 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:52.097787 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:52.097757 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:52.291951 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:52.291916 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:52.512955 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:52.512928 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:53.098060 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:53.097991 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.305108 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:55.305069 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.571365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:55.571278 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:56.859881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:56.858878 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 22:00:56.859881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:56.859131 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" podUID="174885ad-eb95-4eea-87be-75479e6f118b" containerName="maas-api" containerID="cri-o://0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40" gracePeriod=30 Oct 30 22:00:57.066788 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.066750 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:57.289420 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.289383 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 22:00:57.308803 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.308772 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") pod \"174885ad-eb95-4eea-87be-75479e6f118b\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " Oct 30 22:00:57.308947 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.308852 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\" (UniqueName: \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\") pod \"174885ad-eb95-4eea-87be-75479e6f118b\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " Oct 30 22:00:57.309012 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.308954 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\") pod \"174885ad-eb95-4eea-87be-75479e6f118b\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " Oct 30 22:00:57.309073 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.309013 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\") pod \"174885ad-eb95-4eea-87be-75479e6f118b\" (UID: \"174885ad-eb95-4eea-87be-75479e6f118b\") " Oct 30 22:00:57.309456 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.309424 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca" pod "174885ad-eb95-4eea-87be-75479e6f118b" (UID: "174885ad-eb95-4eea-87be-75479e6f118b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:57.311303 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.311281 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs" pod "174885ad-eb95-4eea-87be-75479e6f118b" (UID: "174885ad-eb95-4eea-87be-75479e6f118b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:57.311450 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.311328 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d" pod "174885ad-eb95-4eea-87be-75479e6f118b" (UID: "174885ad-eb95-4eea-87be-75479e6f118b"). InnerVolumeSpecName "kube-api-access-xsm8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:57.311659 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.311644 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls" pod "174885ad-eb95-4eea-87be-75479e6f118b" (UID: "174885ad-eb95-4eea-87be-75479e6f118b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:57.410941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.410862 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:57.410941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.410898 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174885ad-eb95-4eea-87be-75479e6f118b-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:57.410941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.410908 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174885ad-eb95-4eea-87be-75479e6f118b-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:57.410941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.410917 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm8d\" (UniqueName: \"kubernetes.io/projected/174885ad-eb95-4eea-87be-75479e6f118b-kube-api-access-xsm8d\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:57.638299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.638261 4918 generic.go:356] "Generic (PLEG): container finished" podID="174885ad-eb95-4eea-87be-75479e6f118b" containerID="0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40" exitCode=0 Oct 30 22:00:57.638470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.638311 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" Oct 30 22:00:57.638470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.638354 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" event={"ID":"174885ad-eb95-4eea-87be-75479e6f118b","Type":"ContainerDied","Data":"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40"} Oct 30 22:00:57.638470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.638409 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq" event={"ID":"174885ad-eb95-4eea-87be-75479e6f118b","Type":"ContainerDied","Data":"5dbeb84a31e94fc39b681137e5ab91c142de79ab3c494efeaaf6d913a0b60b28"} Oct 30 22:00:57.638470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.638431 4918 scope.go:122] "RemoveContainer" containerID="0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40" Oct 30 22:00:57.649473 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.649456 4918 scope.go:122] "RemoveContainer" containerID="0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40" Oct 30 22:00:57.649758 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:57.649740 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40\": container with ID starting with 0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40 not found: ID does not exist" containerID="0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40" Oct 30 22:00:57.649818 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.649766 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40"} err="failed to get container status \"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40\": rpc error: code = NotFound desc = could not find container \"0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40\": container with ID starting with 0cf9a0cfe1b4ff6ec1e4946dfc5d9abd3414bd88b00187540402149518763d40 not found: ID does not exist" Oct 30 22:00:57.798215 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.798133 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 22:00:57.844027 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:57.843998 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71c0c7-d846bd488-hndbq"] Oct 30 22:00:58.132774 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.131813 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.158183 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.158146 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 22:00:58.158383 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.158357 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" podUID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" containerName="maas-api" containerID="cri-o://0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4" gracePeriod=30 Oct 30 22:00:58.367442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.367404 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.488543 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.488516 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 22:00:58.553456 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.553412 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\" (UniqueName: \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\") pod \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " Oct 30 22:00:58.553663 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.553488 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\") pod \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " Oct 30 22:00:58.553663 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.553581 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\") pod \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " Oct 30 22:00:58.553835 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.553684 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") pod \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\" (UID: \"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1\") " Oct 30 22:00:58.568101 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.559460 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca" pod "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" (UID: "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:58.572063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.572009 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls" pod "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" (UID: "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:58.581987 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.581949 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz" pod "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" (UID: "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1"). InnerVolumeSpecName "kube-api-access-6b8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:58.583123 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.583093 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs" pod "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" (UID: "6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:58.648119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.648033 4918 generic.go:356] "Generic (PLEG): container finished" podID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" containerID="0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4" exitCode=0 Oct 30 22:00:58.648119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.648087 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" Oct 30 22:00:58.648354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.648124 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" event={"ID":"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1","Type":"ContainerDied","Data":"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4"} Oct 30 22:00:58.648354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.648182 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v" event={"ID":"6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1","Type":"ContainerDied","Data":"6b00d864e96c63b561a41edeefd94ebefc02bc90ed60cb5e287b0e74915a915e"} Oct 30 22:00:58.648354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.648214 4918 scope.go:122] "RemoveContainer" containerID="0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4" Oct 30 22:00:58.653005 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.652982 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.655692 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.655664 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.655831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.655696 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.655831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.655727 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8vz\" (UniqueName: \"kubernetes.io/projected/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-kube-api-access-6b8vz\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.655831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.655743 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.659810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.659790 4918 scope.go:122] "RemoveContainer" containerID="0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4" Oct 30 22:00:58.660297 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:00:58.660085 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4\": container with ID starting with 0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4 not found: ID does not exist" containerID="0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4" Oct 30 22:00:58.660297 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.660118 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4"} err="failed to get container status \"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4\": rpc error: code = NotFound desc = could not find container \"0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4\": container with ID starting with 0686c10f195c87310591a4898fae25decc4852770c37f35f38f7d0210ff4c6d4 not found: ID does not exist" Oct 30 22:00:58.824982 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.824950 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 22:00:58.828559 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.828525 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-40bb78-57cfcb55db-vv85v"] Oct 30 22:00:58.969113 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:58.968288 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:59.432347 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:59.432314 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174885ad-eb95-4eea-87be-75479e6f118b" path="/var/lib/kubelet/pods/174885ad-eb95-4eea-87be-75479e6f118b/volumes" Oct 30 22:00:59.432815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:59.432799 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" path="/var/lib/kubelet/pods/6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1/volumes" Oct 30 22:00:59.636224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:00:59.636190 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.040441 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:00.040410 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.291034 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:00.290932 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.488167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:00.488118 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.905446 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:00.905414 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.100400 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.100372 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.269226 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.269153 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 22:01:01.269372 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.269356 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" containerID="cri-o://17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a" gracePeriod=30 Oct 30 22:01:01.284411 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.284382 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.490610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.490573 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.917119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:01.917037 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.328801 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.328762 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.422399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.422356 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" containerID="cri-o://5621eb888415687f777bb964a0a0871fa5ebd7a2b259ee603f7118392c322314" gracePeriod=45 Oct 30 22:01:05.737381 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.737175 4918 generic.go:356] "Generic (PLEG): container finished" podID="89b595f3-10e5-4df8-990c-f49770ba0924" containerID="5621eb888415687f777bb964a0a0871fa5ebd7a2b259ee603f7118392c322314" exitCode=0 Oct 30 22:01:05.737381 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.737326 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" event={"ID":"89b595f3-10e5-4df8-990c-f49770ba0924","Type":"ContainerDied","Data":"5621eb888415687f777bb964a0a0871fa5ebd7a2b259ee603f7118392c322314"} Oct 30 22:01:05.791931 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.791905 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 22:01:05.908601 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908555 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.908814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908688 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.908814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908741 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\" (UniqueName: \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.908814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908785 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.909015 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908814 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.909015 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908892 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\" (UniqueName: \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\") pod \"89b595f3-10e5-4df8-990c-f49770ba0924\" (UID: \"89b595f3-10e5-4df8-990c-f49770ba0924\") " Oct 30 22:01:05.909015 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.908901 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.909194 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.909121 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.909282 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.909220 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.910924 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.910790 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.910924 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.910845 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.910924 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.910860 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.912028 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.911998 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:05.912311 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.912286 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:05.912621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:05.912599 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn" pod "89b595f3-10e5-4df8-990c-f49770ba0924" (UID: "89b595f3-10e5-4df8-990c-f49770ba0924"). InnerVolumeSpecName "kube-api-access-f7btn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:06.011750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.011700 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b595f3-10e5-4df8-990c-f49770ba0924-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.011750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.011747 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b595f3-10e5-4df8-990c-f49770ba0924-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.011750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.011757 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7btn\" (UniqueName: \"kubernetes.io/projected/89b595f3-10e5-4df8-990c-f49770ba0924-kube-api-access-f7btn\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.644293 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.644263 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 22:01:06.725476 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725446 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725511 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725584 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725617 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725644 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725677 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725846 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.725963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725926 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:06.726072 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.725981 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:06.726196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726118 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.726196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726136 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:06.726196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726144 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:06.726196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726186 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\") pod \"66a553b9-fd1d-496b-9134-28b19ca0d850\" (UID: \"66a553b9-fd1d-496b-9134-28b19ca0d850\") " Oct 30 22:01:06.726523 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726493 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:06.726863 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726838 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.726961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726869 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.726961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726884 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.726961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726897 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a553b9-fd1d-496b-9134-28b19ca0d850-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.726961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.726911 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.728185 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.728164 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:01:06.728275 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.728218 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "kube-api-access-zl2lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:06.728275 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.728247 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:06.728369 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.728355 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token" pod "66a553b9-fd1d-496b-9134-28b19ca0d850" (UID: "66a553b9-fd1d-496b-9134-28b19ca0d850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:06.747334 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.747312 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" Oct 30 22:01:06.747423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.747309 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw" event={"ID":"89b595f3-10e5-4df8-990c-f49770ba0924","Type":"ContainerDied","Data":"47a81de3284da1e41e8d9e454aef2c4b6bdf1c1f750fd1428a18d6a7dc20fe79"} Oct 30 22:01:06.747498 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.747450 4918 scope.go:122] "RemoveContainer" containerID="5621eb888415687f777bb964a0a0871fa5ebd7a2b259ee603f7118392c322314" Oct 30 22:01:06.749286 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.749261 4918 generic.go:356] "Generic (PLEG): container finished" podID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerID="17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a" exitCode=0 Oct 30 22:01:06.749388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.749316 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" Oct 30 22:01:06.749388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.749346 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerDied","Data":"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a"} Oct 30 22:01:06.749510 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.749424 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4" event={"ID":"66a553b9-fd1d-496b-9134-28b19ca0d850","Type":"ContainerDied","Data":"c7cd0e0d798c4cb1c4d869d9ec50a9bfa857ced2ba1d546ffba3dc2017e51ac2"} Oct 30 22:01:06.762639 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.762622 4918 scope.go:122] "RemoveContainer" containerID="17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a" Oct 30 22:01:06.773966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.773949 4918 scope.go:122] "RemoveContainer" containerID="9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84" Oct 30 22:01:06.783420 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.783396 4918 scope.go:122] "RemoveContainer" containerID="17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a" Oct 30 22:01:06.783716 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:06.783684 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a\": container with ID starting with 17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a not found: ID does not exist" containerID="17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a" Oct 30 22:01:06.783790 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.783745 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a"} err="failed to get container status \"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a\": rpc error: code = NotFound desc = could not find container \"17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a\": container with ID starting with 17c8222bd903354d89051f57b33455eefe568635abdc2a780a04bfe4cf01687a not found: ID does not exist" Oct 30 22:01:06.783790 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.783769 4918 scope.go:122] "RemoveContainer" containerID="9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84" Oct 30 22:01:06.784029 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:06.784009 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84\": container with ID starting with 9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84 not found: ID does not exist" containerID="9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84" Oct 30 22:01:06.784088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.784037 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84"} err="failed to get container status \"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84\": rpc error: code = NotFound desc = could not find container \"9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84\": container with ID starting with 9bd965b3a960aaaf064478d47711906725d28907679292799485a1e4df83ea84 not found: ID does not exist" Oct 30 22:01:06.828695 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.828666 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2lk\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-kube-api-access-zl2lk\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.828695 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.828694 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.828920 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.828745 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.828920 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.828762 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a553b9-fd1d-496b-9134-28b19ca0d850-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.860510 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.860490 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 22:01:06.924218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.924193 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 22:01:06.928957 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.928931 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2df7b6-openshift-default-58fd6fdc97-447z4"] Oct 30 22:01:06.993716 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:06.993669 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.031069 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.031040 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 22:01:07.036492 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.036467 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-71aab5/distinct-2-w3-71aab5-kserve-7d8c6c6bd6-rgknw"] Oct 30 22:01:07.253873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.253784 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.429474 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.429442 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" path="/var/lib/kubelet/pods/66a553b9-fd1d-496b-9134-28b19ca0d850/volumes" Oct 30 22:01:07.430076 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.430050 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" path="/var/lib/kubelet/pods/89b595f3-10e5-4df8-990c-f49770ba0924/volumes" Oct 30 22:01:07.498945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.498920 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.554289 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.554211 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 22:01:07.554464 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.554440 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" containerID="cri-o://94676b8f1836a4306e0ebccf37d15cd17504ead3378489effb7b91e3a3537519" gracePeriod=30 Oct 30 22:01:07.701542 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:07.701485 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:08.567990 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:08.567954 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:08.812059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:08.812031 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.010537 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:09.008953 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.493744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:10.480084 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.697166 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:10.697135 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.914557 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:10.914519 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.824823 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.824795 4918 generic.go:356] "Generic (PLEG): container finished" podID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerID="94676b8f1836a4306e0ebccf37d15cd17504ead3378489effb7b91e3a3537519" exitCode=0 Oct 30 22:01:12.825156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.824855 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerDied","Data":"94676b8f1836a4306e0ebccf37d15cd17504ead3378489effb7b91e3a3537519"} Oct 30 22:01:12.825156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.824904 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" event={"ID":"be8a9e9d-36f9-40a8-a031-34907e07baaf","Type":"ContainerDied","Data":"85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95"} Oct 30 22:01:12.825156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.824924 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85751682bc75e313ee9e814950bdcc248f406200eea8df3b3a61c41e41cffa95" Oct 30 22:01:12.825156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.824937 4918 scope.go:122] "RemoveContainer" containerID="8decb4ed657a512d3a058ab20701254257788d1e234e463d301f0268b40a1f03" Oct 30 22:01:12.841612 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:12.841582 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 22:01:13.029263 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029230 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029273 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029307 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029324 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029372 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029397 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029427 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029570 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029603 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\") pod \"be8a9e9d-36f9-40a8-a031-34907e07baaf\" (UID: \"be8a9e9d-36f9-40a8-a031-34907e07baaf\") " Oct 30 22:01:13.029798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029661 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:13.029798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029698 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:13.030071 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.029912 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:13.030170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.030144 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:13.030170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.030155 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:13.030295 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.030251 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.030295 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.030291 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.030391 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.030305 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.032040 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.032016 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:13.032251 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.032226 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:01:13.032550 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.032526 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:13.032619 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.032563 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r" pod "be8a9e9d-36f9-40a8-a031-34907e07baaf" (UID: "be8a9e9d-36f9-40a8-a031-34907e07baaf"). InnerVolumeSpecName "kube-api-access-z9l9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:13.131236 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131201 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.131236 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131233 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9l9r\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-kube-api-access-z9l9r\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.131410 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131248 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.131410 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131261 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.131410 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131272 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8a9e9d-36f9-40a8-a031-34907e07baaf-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.131410 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.131284 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8a9e9d-36f9-40a8-a031-34907e07baaf-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.836016 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.835942 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss" Oct 30 22:01:13.893938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:13.893907 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:14.087220 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.087124 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:14.123737 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.123691 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 22:01:14.132665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.132641 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-71c0c7-openshift-default-5b95f86b7f-brnss"] Oct 30 22:01:14.269674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.269630 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 22:01:14.270002 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.269971 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerName="discovery" containerID="cri-o://235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087" gracePeriod=30 Oct 30 22:01:14.585114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.585085 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 22:01:14.656208 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656136 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656208 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656206 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656392 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656258 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656392 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656317 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656478 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656389 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656551 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656520 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656616 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656568 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\") pod \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\" (UID: \"417f64fe-3c08-4fd3-9d06-52b3a278ae94\") " Oct 30 22:01:14.656973 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.656945 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:14.657164 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.657139 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-ca-configmap\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.658819 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.658790 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "kube-api-access-bm7ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:14.659074 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.658969 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:14.659074 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.658997 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:14.659074 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.659027 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:14.659212 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.659115 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:14.659525 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.659491 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert" pod "417f64fe-3c08-4fd3-9d06-52b3a278ae94" (UID: "417f64fe-3c08-4fd3-9d06-52b3a278ae94"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:14.757810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757777 4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-cacerts\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.757810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757804 4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-kubeconfig\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.757810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757814 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-csr-dns-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.758035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757823 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm7ck\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-kube-api-access-bm7ck\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.758035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757832 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f64fe-3c08-4fd3-9d06-52b3a278ae94-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.758035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.757841 4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f64fe-3c08-4fd3-9d06-52b3a278ae94-local-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:14.846949 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.846910 4918 generic.go:356] "Generic (PLEG): container finished" podID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerID="235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087" exitCode=0 Oct 30 22:01:14.847406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.846982 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" Oct 30 22:01:14.847406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.847004 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" event={"ID":"417f64fe-3c08-4fd3-9d06-52b3a278ae94","Type":"ContainerDied","Data":"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087"} Oct 30 22:01:14.847406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.847098 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt" event={"ID":"417f64fe-3c08-4fd3-9d06-52b3a278ae94","Type":"ContainerDied","Data":"c74ec5b6ca4efec4897c6c738e2c4d793a109ddf97800269356fc19efeac8340"} Oct 30 22:01:14.847406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.847126 4918 scope.go:122] "RemoveContainer" containerID="235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087" Oct 30 22:01:14.864467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.864447 4918 scope.go:122] "RemoveContainer" containerID="235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087" Oct 30 22:01:14.864782 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:14.864748 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087\": container with ID starting with 235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087 not found: ID does not exist" containerID="235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087" Oct 30 22:01:14.864873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:14.864791 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087"} err="failed to get container status \"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087\": rpc error: code = NotFound desc = could not find container \"235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087\": container with ID starting with 235e7abe7ae65628989958b0130ceffef023847e8010213d7e5f417bf7abf087 not found: ID does not exist" Oct 30 22:01:15.052815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:15.052470 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 22:01:15.058141 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:15.058112 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z6ltt"] Oct 30 22:01:15.429812 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:15.429780 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" path="/var/lib/kubelet/pods/417f64fe-3c08-4fd3-9d06-52b3a278ae94/volumes" Oct 30 22:01:15.430310 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:15.430295 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" path="/var/lib/kubelet/pods/be8a9e9d-36f9-40a8-a031-34907e07baaf/volumes" Oct 30 22:01:15.537304 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:15.537274 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:15.804116 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:15.804025 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:01:20.330841 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:20.330794 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.307426 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:21.307388 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.562388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:21.562305 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.883108 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:21.883038 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" containerID="cri-o://0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42" gracePeriod=45 Oct 30 22:01:22.168052 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.168027 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 22:01:22.240693 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240662 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.240849 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240839 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.240912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240895 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.240969 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240926 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.240969 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240947 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\" (UniqueName: \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.241087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.240971 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.241087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241022 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\") pod \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\" (UID: \"05b0c6cb-bd2f-49e6-ba66-dc39749770bc\") " Oct 30 22:01:22.241087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241073 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.241234 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241132 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.241429 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241410 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.241495 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241436 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.241495 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.241450 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.243247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.243219 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.243353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.243263 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "kube-api-access-rc2mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:22.243353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.243306 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs" pod "05b0c6cb-bd2f-49e6-ba66-dc39749770bc" (UID: "05b0c6cb-bd2f-49e6-ba66-dc39749770bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:22.342469 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.342433 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2mm\" (UniqueName: \"kubernetes.io/projected/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-kube-api-access-rc2mm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.342469 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.342472 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.342655 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.342487 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b0c6cb-bd2f-49e6-ba66-dc39749770bc-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.907566 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.907530 4918 generic.go:356] "Generic (PLEG): container finished" podID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerID="0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42" exitCode=0 Oct 30 22:01:22.908063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.907604 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" Oct 30 22:01:22.908063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.907636 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" event={"ID":"05b0c6cb-bd2f-49e6-ba66-dc39749770bc","Type":"ContainerDied","Data":"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42"} Oct 30 22:01:22.908063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.907681 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw" event={"ID":"05b0c6cb-bd2f-49e6-ba66-dc39749770bc","Type":"ContainerDied","Data":"d2076bc9808c67de9ec45bc65633097998aaaa495e053a8a66b6b3d060b438bb"} Oct 30 22:01:22.908063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.907720 4918 scope.go:122] "RemoveContainer" containerID="0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42" Oct 30 22:01:22.927266 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.927247 4918 scope.go:122] "RemoveContainer" containerID="0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42" Oct 30 22:01:22.927903 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:22.927870 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42\": container with ID starting with 0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42 not found: ID does not exist" containerID="0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42" Oct 30 22:01:22.928030 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:22.927929 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42"} err="failed to get container status \"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42\": rpc error: code = NotFound desc = could not find container \"0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42\": container with ID starting with 0e12776e39334fcb4e608046d767d8ba4e94a357cd3a3122df77a8968a729f42 not found: ID does not exist" Oct 30 22:01:23.067973 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.067943 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 22:01:23.114125 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.114089 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-2-w1-19dd4f-kserve-7b9f6677c9-4xnnw"] Oct 30 22:01:23.430227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.430188 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" path="/var/lib/kubelet/pods/05b0c6cb-bd2f-49e6-ba66-dc39749770bc/volumes" Oct 30 22:01:23.445360 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.445329 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 22:01:23.559804 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.559774 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 22:01:23.859937 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:23.859893 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:24.049687 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.049641 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:24.288955 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.288885 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 22:01:24.289107 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.289077 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" podUID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" containerName="maas-api" containerID="cri-o://3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52" gracePeriod=30 Oct 30 22:01:24.480326 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.480279 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:24.562313 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.562288 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 22:01:24.673288 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.673257 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\") pod \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " Oct 30 22:01:24.673481 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.673403 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\") pod \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " Oct 30 22:01:24.673481 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.673451 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\" (UniqueName: \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\") pod \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " Oct 30 22:01:24.673601 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.673524 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") pod \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\" (UID: \"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590\") " Oct 30 22:01:24.673830 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.673773 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca" pod "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" (UID: "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:24.674399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.674377 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.675878 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.675849 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8" pod "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" (UID: "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590"). InnerVolumeSpecName "kube-api-access-f77j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:24.675878 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.675857 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls" pod "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" (UID: "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:24.676018 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.675893 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs" pod "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" (UID: "8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:24.775512 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.775481 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77j8\" (UniqueName: \"kubernetes.io/projected/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-kube-api-access-f77j8\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.775512 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.775510 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.775978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.775523 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.927677 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.927643 4918 generic.go:356] "Generic (PLEG): container finished" podID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" containerID="3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52" exitCode=0 Oct 30 22:01:24.927912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.927732 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" Oct 30 22:01:24.927912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.927728 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" event={"ID":"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590","Type":"ContainerDied","Data":"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52"} Oct 30 22:01:24.927912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.927863 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276" event={"ID":"8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590","Type":"ContainerDied","Data":"155892fca5236c35430219ef7167786217cff07408c136cc603a4e6e9a030342"} Oct 30 22:01:24.927912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.927892 4918 scope.go:122] "RemoveContainer" containerID="3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52" Oct 30 22:01:24.943061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.943036 4918 scope.go:122] "RemoveContainer" containerID="3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52" Oct 30 22:01:24.943383 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:24.943349 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52\": container with ID starting with 3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52 not found: ID does not exist" containerID="3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52" Oct 30 22:01:24.943474 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:24.943386 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52"} err="failed to get container status \"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52\": rpc error: code = NotFound desc = could not find container \"3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52\": container with ID starting with 3f7037bc8e8823163dbd3d625bd7f7c82e5b4416b2ca56b8a075a2b5e203da52 not found: ID does not exist" Oct 30 22:01:25.090205 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:25.090167 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 22:01:25.111361 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:25.111323 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfdf30b9-7d4557498b-hf276"] Oct 30 22:01:25.435863 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:25.435825 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" path="/var/lib/kubelet/pods/8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590/volumes" Oct 30 22:01:26.009633 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:26.009603 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.234133 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:26.234102 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.438683 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:26.438648 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.018784 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:27.018754 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:28.185792 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.184269 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:28.372768 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.372733 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:28.445823 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.445742 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" podUID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" containerName="payload-processing" containerID="cri-o://9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c" gracePeriod=25 Oct 30 22:01:28.560494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.560465 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" podUID="4faeebe6-b203-4144-955c-914bf5a968e4" containerName="payload-pre-processing" containerID="cri-o://9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb" gracePeriod=25 Oct 30 22:01:28.725087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.725054 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 22:01:28.828183 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.828155 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\") pod \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " Oct 30 22:01:28.828349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.828304 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\" (UniqueName: \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\") pod \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\" (UID: \"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e\") " Oct 30 22:01:28.828566 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.828537 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume" pod "10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" (UID: "10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:28.828874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.828853 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.830345 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.830322 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66" pod "10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" (UID: "10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e"). InnerVolumeSpecName "kube-api-access-5nd66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:28.837112 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.837096 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 22:01:28.929374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.929339 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\" (UniqueName: \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\") pod \"4faeebe6-b203-4144-955c-914bf5a968e4\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " Oct 30 22:01:28.929554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.929534 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\") pod \"4faeebe6-b203-4144-955c-914bf5a968e4\" (UID: \"4faeebe6-b203-4144-955c-914bf5a968e4\") " Oct 30 22:01:28.929922 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.929898 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume" pod "4faeebe6-b203-4144-955c-914bf5a968e4" (UID: "4faeebe6-b203-4144-955c-914bf5a968e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:28.930121 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.930098 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nd66\" (UniqueName: \"kubernetes.io/projected/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e-kube-api-access-5nd66\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.930189 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.930127 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4faeebe6-b203-4144-955c-914bf5a968e4-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:28.931537 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.931519 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc" pod "4faeebe6-b203-4144-955c-914bf5a968e4" (UID: "4faeebe6-b203-4144-955c-914bf5a968e4"). InnerVolumeSpecName "kube-api-access-ffgrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:28.961950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.961883 4918 generic.go:356] "Generic (PLEG): container finished" podID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" containerID="9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c" exitCode=0 Oct 30 22:01:28.962085 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.962018 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" event={"ID":"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e","Type":"ContainerDied","Data":"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c"} Oct 30 22:01:28.962085 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.962067 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" event={"ID":"10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e","Type":"ContainerDied","Data":"eb563270f0635e3393806c48f0a0c88669f9f639311d686185f25dcacaae0e57"} Oct 30 22:01:28.962182 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.962097 4918 scope.go:122] "RemoveContainer" containerID="9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c" Oct 30 22:01:28.962256 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.962242 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl" Oct 30 22:01:28.969149 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.969126 4918 generic.go:356] "Generic (PLEG): container finished" podID="4faeebe6-b203-4144-955c-914bf5a968e4" containerID="9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb" exitCode=0 Oct 30 22:01:28.969244 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.969189 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" event={"ID":"4faeebe6-b203-4144-955c-914bf5a968e4","Type":"ContainerDied","Data":"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb"} Oct 30 22:01:28.969244 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.969223 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" event={"ID":"4faeebe6-b203-4144-955c-914bf5a968e4","Type":"ContainerDied","Data":"59deda55ce8ab8816da802e5443c5e032c219ff8ce72c4bab7d4d2bd14e265ac"} Oct 30 22:01:28.969325 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.969276 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k" Oct 30 22:01:28.982385 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.982347 4918 scope.go:122] "RemoveContainer" containerID="9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c" Oct 30 22:01:28.982669 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:28.982645 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c\": container with ID starting with 9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c not found: ID does not exist" containerID="9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c" Oct 30 22:01:28.982767 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.982680 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c"} err="failed to get container status \"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c\": rpc error: code = NotFound desc = could not find container \"9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c\": container with ID starting with 9277e7c03afaba3cff4ba9eeaecd346fb008d1012b1d0e303ee06f943595fe2c not found: ID does not exist" Oct 30 22:01:28.982767 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.982730 4918 scope.go:122] "RemoveContainer" containerID="9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb" Oct 30 22:01:28.994342 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.994322 4918 scope.go:122] "RemoveContainer" containerID="9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb" Oct 30 22:01:28.994683 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:28.994659 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb\": container with ID starting with 9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb not found: ID does not exist" containerID="9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb" Oct 30 22:01:28.994814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:28.994689 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb"} err="failed to get container status \"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb\": rpc error: code = NotFound desc = could not find container \"9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb\": container with ID starting with 9b662b99e84dcb980a66f13c2ec07dc818c1dd9dae66a2f034e95dc8aca5ebeb not found: ID does not exist" Oct 30 22:01:29.031788 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.031763 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgrc\" (UniqueName: \"kubernetes.io/projected/4faeebe6-b203-4144-955c-914bf5a968e4-kube-api-access-ffgrc\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:29.129791 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.129731 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 22:01:29.133423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.133396 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ca0a55-5865d7d47f-sp99k"] Oct 30 22:01:29.237540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.237463 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 22:01:29.241469 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.241444 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ca0a55-5f576f5b74-xkgbl"] Oct 30 22:01:29.429564 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.429532 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" path="/var/lib/kubelet/pods/10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e/volumes" Oct 30 22:01:29.429988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:29.429970 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4faeebe6-b203-4144-955c-914bf5a968e4" path="/var/lib/kubelet/pods/4faeebe6-b203-4144-955c-914bf5a968e4/volumes" Oct 30 22:01:30.450407 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:30.450350 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.388456 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:31.388418 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.027225 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:32.027193 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.314218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:36.314186 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.514092 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:36.514045 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:41.043449 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:41.043407 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:44.032153 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:44.032110 4918 scope.go:122] "RemoveContainer" containerID="94676b8f1836a4306e0ebccf37d15cd17504ead3378489effb7b91e3a3537519" Oct 30 22:01:45.359383 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.359341 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 22:01:45.359902 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.359604 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" podUID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" containerName="maas-api" containerID="cri-o://a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9" gracePeriod=30 Oct 30 22:01:45.594185 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.594151 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:45.656513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.656485 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 22:01:45.814625 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:45.814582 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:01:45.825982 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.825954 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:45.832088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.832067 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") pod \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " Oct 30 22:01:45.832227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.832118 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\") pod \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " Oct 30 22:01:45.832227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.832149 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\") pod \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " Oct 30 22:01:45.832227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.832175 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\" (UniqueName: \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\") pod \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\" (UID: \"2aeec501-49d9-44a1-bb78-ae7ac11a65a2\") " Oct 30 22:01:45.832612 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.832578 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca" pod "2aeec501-49d9-44a1-bb78-ae7ac11a65a2" (UID: "2aeec501-49d9-44a1-bb78-ae7ac11a65a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:45.834311 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.834280 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls" pod "2aeec501-49d9-44a1-bb78-ae7ac11a65a2" (UID: "2aeec501-49d9-44a1-bb78-ae7ac11a65a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:45.834437 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.834413 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx" pod "2aeec501-49d9-44a1-bb78-ae7ac11a65a2" (UID: "2aeec501-49d9-44a1-bb78-ae7ac11a65a2"). InnerVolumeSpecName "kube-api-access-5q6gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:45.834505 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.834487 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs" pod "2aeec501-49d9-44a1-bb78-ae7ac11a65a2" (UID: "2aeec501-49d9-44a1-bb78-ae7ac11a65a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:45.933573 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.933487 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:45.933573 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.933525 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:45.933573 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.933540 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6gx\" (UniqueName: \"kubernetes.io/projected/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-kube-api-access-5q6gx\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:45.933573 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:45.933556 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aeec501-49d9-44a1-bb78-ae7ac11a65a2-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:46.041958 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.041923 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:46.095868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.095834 4918 generic.go:356] "Generic (PLEG): container finished" podID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" containerID="a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9" exitCode=0 Oct 30 22:01:46.096038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.095883 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" Oct 30 22:01:46.096038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.095913 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" event={"ID":"2aeec501-49d9-44a1-bb78-ae7ac11a65a2","Type":"ContainerDied","Data":"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9"} Oct 30 22:01:46.096038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.095967 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft" event={"ID":"2aeec501-49d9-44a1-bb78-ae7ac11a65a2","Type":"ContainerDied","Data":"00834e83fa32f5e761a5205eb7e735def7cee2a2cd5fc7007d82fd755a63c920"} Oct 30 22:01:46.096038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.095989 4918 scope.go:122] "RemoveContainer" containerID="a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9" Oct 30 22:01:46.107234 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.107213 4918 scope.go:122] "RemoveContainer" containerID="a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9" Oct 30 22:01:46.107514 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:46.107489 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9\": container with ID starting with a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9 not found: ID does not exist" containerID="a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9" Oct 30 22:01:46.107609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.107519 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9"} err="failed to get container status \"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9\": rpc error: code = NotFound desc = could not find container \"a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9\": container with ID starting with a73ff7187d519d181f66c0c44bf2613d24be518f2f41b401c0abbadb3f54fea9 not found: ID does not exist" Oct 30 22:01:46.336491 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.336400 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 22:01:46.342000 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.341970 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e874-66c758b5f-pqtft"] Oct 30 22:01:46.422123 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:46.422088 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:47.059204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:47.059173 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:47.359061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:47.359029 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:47.428296 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:47.428265 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" path="/var/lib/kubelet/pods/2aeec501-49d9-44a1-bb78-ae7ac11a65a2/volumes" Oct 30 22:01:47.605128 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:47.605090 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:47.911452 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:47.911422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.113812 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:48.113781 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:49.739880 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:49.739848 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:50.198228 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:50.198173 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:50.414786 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:50.414754 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:52.769258 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:52.769225 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.673588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:54.673557 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.924101 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:54.923999 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.126072 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:55.126016 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.313004 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:55.312921 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.897928 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:55.897889 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.088487 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.088446 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 22:01:57.088948 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.088700 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" podUID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" containerName="maas-api" containerID="cri-o://bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29" gracePeriod=30 Oct 30 22:01:57.157514 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.157467 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.365759 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.365726 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.370252 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.370225 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 22:01:57.463794 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.463765 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\" (UniqueName: \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\") pod \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " Oct 30 22:01:57.463944 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.463820 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") pod \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " Oct 30 22:01:57.463944 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.463920 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\") pod \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " Oct 30 22:01:57.464050 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.464005 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\") pod \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\" (UID: \"f7591d4f-54e9-4a96-b72a-8a4ec57af991\") " Oct 30 22:01:57.464308 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.464280 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca" pod "f7591d4f-54e9-4a96-b72a-8a4ec57af991" (UID: "f7591d4f-54e9-4a96-b72a-8a4ec57af991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:57.464470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.464446 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7591d4f-54e9-4a96-b72a-8a4ec57af991-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:57.466035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.466013 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z" pod "f7591d4f-54e9-4a96-b72a-8a4ec57af991" (UID: "f7591d4f-54e9-4a96-b72a-8a4ec57af991"). InnerVolumeSpecName "kube-api-access-gr29z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:57.466135 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.466113 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls" pod "f7591d4f-54e9-4a96-b72a-8a4ec57af991" (UID: "f7591d4f-54e9-4a96-b72a-8a4ec57af991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:57.466189 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.466138 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs" pod "f7591d4f-54e9-4a96-b72a-8a4ec57af991" (UID: "f7591d4f-54e9-4a96-b72a-8a4ec57af991"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:57.565943 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.565912 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:57.565943 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.565939 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr29z\" (UniqueName: \"kubernetes.io/projected/f7591d4f-54e9-4a96-b72a-8a4ec57af991-kube-api-access-gr29z\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:57.565943 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:57.565950 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7591d4f-54e9-4a96-b72a-8a4ec57af991-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:58.120349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.120310 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:58.185115 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.185083 4918 generic.go:356] "Generic (PLEG): container finished" podID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" containerID="bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29" exitCode=0 Oct 30 22:01:58.185299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.185130 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" Oct 30 22:01:58.185299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.185163 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" event={"ID":"f7591d4f-54e9-4a96-b72a-8a4ec57af991","Type":"ContainerDied","Data":"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29"} Oct 30 22:01:58.185299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.185217 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg" event={"ID":"f7591d4f-54e9-4a96-b72a-8a4ec57af991","Type":"ContainerDied","Data":"057ee076c1c3360a722f31d38546fefed28126e259dc47d37606ec5efe56b58c"} Oct 30 22:01:58.185299 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.185242 4918 scope.go:122] "RemoveContainer" containerID="bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29" Oct 30 22:01:58.198859 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.198843 4918 scope.go:122] "RemoveContainer" containerID="bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29" Oct 30 22:01:58.199199 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:01:58.199179 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29\": container with ID starting with bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29 not found: ID does not exist" containerID="bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29" Oct 30 22:01:58.199279 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.199213 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29"} err="failed to get container status \"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29\": rpc error: code = NotFound desc = could not find container \"bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29\": container with ID starting with bae4bf762494b1ee633e5f9df44539ee5b799b1cc2b51bd0bf1b8d7904f17d29 not found: ID does not exist" Oct 30 22:01:58.346590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.346552 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 22:01:58.350920 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.350896 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ca0a55-847d8db6df-79khg"] Oct 30 22:01:58.883049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:58.883018 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.109910 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:59.109878 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.316042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:59.315966 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.428230 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:59.428198 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" path="/var/lib/kubelet/pods/f7591d4f-54e9-4a96-b72a-8a4ec57af991/volumes" Oct 30 22:01:59.971351 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:01:59.971322 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:01.026123 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:01.026074 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:02.834342 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:02.834296 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:03.091215 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:03.091124 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:04.403986 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:04.403948 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:06.210027 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.209992 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:06.495908 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.495821 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:06.746684 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.746603 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:02:06.747250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747230 4918 cpu_manager.go:414] "containerMap: removing container" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747252 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747271 4918 cpu_manager.go:414] "containerMap: removing container" podUID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" containerName="payload-processing" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747278 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" containerName="payload-processing" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747297 4918 cpu_manager.go:414] "containerMap: removing container" podUID="174885ad-eb95-4eea-87be-75479e6f118b" containerName="maas-api" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747304 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174885ad-eb95-4eea-87be-75479e6f118b" containerName="maas-api" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747320 4918 cpu_manager.go:414] "containerMap: removing container" podUID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" containerName="maas-api" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747328 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" containerName="maas-api" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747341 4918 cpu_manager.go:414] "containerMap: removing container" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747348 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747359 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4faeebe6-b203-4144-955c-914bf5a968e4" containerName="payload-pre-processing" Oct 30 22:02:06.747365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747367 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4faeebe6-b203-4144-955c-914bf5a968e4" containerName="payload-pre-processing" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747406 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747415 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747431 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747438 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747449 4918 cpu_manager.go:414] "containerMap: removing container" podUID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747457 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747470 4918 cpu_manager.go:414] "containerMap: removing container" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747477 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747490 4918 cpu_manager.go:414] "containerMap: removing container" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747497 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747508 4918 cpu_manager.go:414] "containerMap: removing container" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerName="discovery" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747515 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerName="discovery" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747535 4918 cpu_manager.go:414] "containerMap: removing container" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747543 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747678 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747689 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b0c6cb-bd2f-49e6-ba66-dc39749770bc" containerName="main" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747697 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="10adf94a-3b9a-45bc-aca9-bbb9f45b0f1e" containerName="payload-processing" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747728 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="174885ad-eb95-4eea-87be-75479e6f118b" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747738 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6684ee9f-a76a-4cf0-8c4f-f9539c6ee0d1" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747747 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747756 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4faeebe6-b203-4144-955c-914bf5a968e4" containerName="payload-pre-processing" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747771 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7591d4f-54e9-4a96-b72a-8a4ec57af991" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747782 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aeec501-49d9-44a1-bb78-ae7ac11a65a2" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747790 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc01bf5-b3b5-4336-b6b7-9cfb7fb4a590" containerName="maas-api" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747801 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8a9e9d-36f9-40a8-a031-34907e07baaf" containerName="istio-proxy" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747811 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b595f3-10e5-4df8-990c-f49770ba0924" containerName="main" Oct 30 22:02:06.747954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.747821 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="417f64fe-3c08-4fd3-9d06-52b3a278ae94" containerName="discovery" Oct 30 22:02:06.753383 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.753362 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:06.809837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.809810 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:02:06.851106 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.851073 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-87eb6a-dockercfg-z86gl\"" Oct 30 22:02:06.851450 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.851400 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-87eb6a\"" Oct 30 22:02:06.893728 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.893570 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6f5\" (UniqueName: \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:06.893931 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.893871 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:06.975968 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.975934 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:02:06.976519 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.976506 4918 cpu_manager.go:414] "containerMap: removing container" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.976583 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.976519 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.976643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.976617 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a553b9-fd1d-496b-9134-28b19ca0d850" containerName="istio-proxy" Oct 30 22:02:06.980484 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.980462 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:06.995425 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.995405 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:06.995540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.995467 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6f5\" (UniqueName: \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:06.996097 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:06.996074 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:07.038816 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.038761 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:02:07.053901 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.053870 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6f5\" (UniqueName: \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\") pod \"payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:07.068499 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.068476 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:07.096541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.096514 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cb7\" (UniqueName: \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.096666 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.096548 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.198118 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.198081 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cb7\" (UniqueName: \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.198318 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.198126 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.198944 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.198918 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.259793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.259766 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cb7\" (UniqueName: \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\") pod \"payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.294393 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.294340 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:07.444113 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.444083 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:02:07.445114 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:02:07.445085 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe56b5a6_df99_4a59_a2fe_f2ef5dcc0bd4.slice/crio-83db4a7fa4f091307fd7b9654aa3788b13490c7e6f0c0b3fb47edd52163c2cd0 WatchSource:0}: Error finding container 83db4a7fa4f091307fd7b9654aa3788b13490c7e6f0c0b3fb47edd52163c2cd0: Status 404 returned error can't find the container with id 83db4a7fa4f091307fd7b9654aa3788b13490c7e6f0c0b3fb47edd52163c2cd0 Oct 30 22:02:07.661210 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.661180 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 22:02:07.674562 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:07.674535 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:02:07.676524 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:02:07.676499 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e753ee_db7c_44a5_ac63_a9416dcd979a.slice/crio-afb9d247c0ca8411b5470af76b44c084639ec3cf741157390201e8d7cfc34db8 WatchSource:0}: Error finding container afb9d247c0ca8411b5470af76b44c084639ec3cf741157390201e8d7cfc34db8: Status 404 returned error can't find the container with id afb9d247c0ca8411b5470af76b44c084639ec3cf741157390201e8d7cfc34db8 Oct 30 22:02:08.062806 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.062689 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:08.255511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.255457 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" event={"ID":"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4","Type":"ContainerStarted","Data":"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b"} Oct 30 22:02:08.255870 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.255826 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" event={"ID":"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4","Type":"ContainerStarted","Data":"83db4a7fa4f091307fd7b9654aa3788b13490c7e6f0c0b3fb47edd52163c2cd0"} Oct 30 22:02:08.255870 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.255871 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:08.257409 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.257388 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" event={"ID":"10e753ee-db7c-44a5-ac63-a9416dcd979a","Type":"ContainerStarted","Data":"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e"} Oct 30 22:02:08.257499 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.257427 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" event={"ID":"10e753ee-db7c-44a5-ac63-a9416dcd979a","Type":"ContainerStarted","Data":"afb9d247c0ca8411b5470af76b44c084639ec3cf741157390201e8d7cfc34db8"} Oct 30 22:02:08.257539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.257528 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:08.325873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.325803 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:08.373988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.373932 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" podStartSLOduration=2.373918663 podStartE2EDuration="2.373918663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:08.371870067 +0000 UTC m=+2909.065352313" watchObservedRunningTime="2026-10-30 22:02:08.373918663 +0000 UTC m=+2909.067400898" Oct 30 22:02:08.489867 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.489801 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" podStartSLOduration=2.489781805 podStartE2EDuration="2.489781805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:08.485366268 +0000 UTC m=+2909.178848507" watchObservedRunningTime="2026-10-30 22:02:08.489781805 +0000 UTC m=+2909.183264043" Oct 30 22:02:08.557525 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.557494 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:08.803363 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:08.803315 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.189554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:09.189519 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.397664 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:09.397631 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.712372 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:09.712329 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:11.018414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:11.018374 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:12.662435 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:12.662393 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" containerID="cri-o://e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3" gracePeriod=25 Oct 30 22:02:14.136245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.136206 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 22:02:14.238978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.238935 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 22:02:14.239212 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.239183 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" podUID="f21fca56-4a41-4e0a-880e-7380c80c2219" containerName="maas-api" containerID="cri-o://f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499" gracePeriod=30 Oct 30 22:02:14.305439 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.305400 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.498667 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.498631 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.534384 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.534358 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 22:02:14.608535 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.608503 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\") pod \"f21fca56-4a41-4e0a-880e-7380c80c2219\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " Oct 30 22:02:14.608743 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.608619 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\") pod \"f21fca56-4a41-4e0a-880e-7380c80c2219\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " Oct 30 22:02:14.608743 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.608666 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\" (UniqueName: \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\") pod \"f21fca56-4a41-4e0a-880e-7380c80c2219\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " Oct 30 22:02:14.608886 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.608864 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") pod \"f21fca56-4a41-4e0a-880e-7380c80c2219\" (UID: \"f21fca56-4a41-4e0a-880e-7380c80c2219\") " Oct 30 22:02:14.609315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.609287 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca" pod "f21fca56-4a41-4e0a-880e-7380c80c2219" (UID: "f21fca56-4a41-4e0a-880e-7380c80c2219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:14.609647 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.609625 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21fca56-4a41-4e0a-880e-7380c80c2219-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:14.611795 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.611765 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs" pod "f21fca56-4a41-4e0a-880e-7380c80c2219" (UID: "f21fca56-4a41-4e0a-880e-7380c80c2219"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:14.612117 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.612086 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg" pod "f21fca56-4a41-4e0a-880e-7380c80c2219" (UID: "f21fca56-4a41-4e0a-880e-7380c80c2219"). InnerVolumeSpecName "kube-api-access-wrpwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:14.612209 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.612172 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls" pod "f21fca56-4a41-4e0a-880e-7380c80c2219" (UID: "f21fca56-4a41-4e0a-880e-7380c80c2219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:14.711008 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.710930 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:14.711008 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.710966 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21fca56-4a41-4e0a-880e-7380c80c2219-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:14.711008 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:14.710981 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrpwg\" (UniqueName: \"kubernetes.io/projected/f21fca56-4a41-4e0a-880e-7380c80c2219-kube-api-access-wrpwg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:15.331245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.331202 4918 generic.go:356] "Generic (PLEG): container finished" podID="f21fca56-4a41-4e0a-880e-7380c80c2219" containerID="f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499" exitCode=0 Oct 30 22:02:15.331793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.331296 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" event={"ID":"f21fca56-4a41-4e0a-880e-7380c80c2219","Type":"ContainerDied","Data":"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499"} Oct 30 22:02:15.331793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.331367 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" event={"ID":"f21fca56-4a41-4e0a-880e-7380c80c2219","Type":"ContainerDied","Data":"292ddd0a98d6dc6df2f7d4ff0fc5e8daa559c89982e289d7da22f1b1a24fde59"} Oct 30 22:02:15.331793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.331398 4918 scope.go:122] "RemoveContainer" containerID="f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499" Oct 30 22:02:15.331793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.331531 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn" Oct 30 22:02:15.350482 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.350464 4918 scope.go:122] "RemoveContainer" containerID="f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499" Oct 30 22:02:15.350778 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:15.350758 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499\": container with ID starting with f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499 not found: ID does not exist" containerID="f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499" Oct 30 22:02:15.350844 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.350786 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499"} err="failed to get container status \"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499\": rpc error: code = NotFound desc = could not find container \"f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499\": container with ID starting with f62c548d18c525a2ce0f065400098c2dcfd1bdab91a8ac0a14fdd9c05ff31499 not found: ID does not exist" Oct 30 22:02:15.494006 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.493966 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 22:02:15.541546 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.541510 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71aab5-594c8dcb8f-pgxbn"] Oct 30 22:02:15.607347 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:15.607269 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.810540 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:15.810501 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:02:16.035326 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:16.035287 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:16.370846 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:16.370807 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:17.062531 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:17.062498 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:17.316883 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:17.316794 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:17.429901 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:17.429861 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21fca56-4a41-4e0a-880e-7380c80c2219" path="/var/lib/kubelet/pods/f21fca56-4a41-4e0a-880e-7380c80c2219/volumes" Oct 30 22:02:17.617551 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:17.617515 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:17.899392 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:17.899302 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:18.284939 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:18.284857 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.267221 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.267188 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:02:19.267634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.267606 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:02:19.526590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.526491 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.720630 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.720561 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.829351 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.829277 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:02:19.829529 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.829498 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" containerID="cri-o://6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0" gracePeriod=30 Oct 30 22:02:19.832781 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.832757 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:02:19.832978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.832951 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" containerID="cri-o://a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6" gracePeriod=30 Oct 30 22:02:19.835732 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.835696 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:02:19.835899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:19.835879 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" containerID="cri-o://eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214" gracePeriod=30 Oct 30 22:02:20.013331 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:20.013289 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:21.285253 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:21.285217 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 22:02:21.637656 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:21.637621 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:21.778540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:21.778498 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.180745 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:22.180690 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.746407 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:22.746380 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.309144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:23.309115 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.518281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:23.518231 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:24.896297 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:24.896271 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.092611 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.092539 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.141246 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.141222 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:02:25.209056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.205613 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:02:25.209056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.207855 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:02:25.264584 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264557 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.264763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264618 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.264763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264670 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.264899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264798 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.264899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264860 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.264899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264861 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.265056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264899 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.265056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.264949 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.265056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265048 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.265227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265098 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\") pod \"c06501d6-4873-4e09-a1c5-0620ba89da6d\" (UID: \"c06501d6-4873-4e09-a1c5-0620ba89da6d\") " Oct 30 22:02:25.265227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265135 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.265227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265147 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.265392 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265360 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.265544 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265522 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.265640 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265616 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.265640 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265631 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.265640 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265640 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.265790 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.265650 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.267469 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.267444 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.267572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.267492 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.267572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.267512 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "kube-api-access-gmwp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.267572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.267493 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo" pod "c06501d6-4873-4e09-a1c5-0620ba89da6d" (UID: "c06501d6-4873-4e09-a1c5-0620ba89da6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.310143 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.310036 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.367846 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.367365 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.368029 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.367978 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.368426 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.368888 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.368921 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.369573 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.369655 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.369988 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.370213 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.370262 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.370309 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.370692 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371362 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371406 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371489 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371616 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371674 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371724 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371757 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371836 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\") pod \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\" (UID: \"b8244505-8b0c-4f00-aeca-abf790ed1d7d\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.371905 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\") pod \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\" (UID: \"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec\") " Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372591 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwp8\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-kube-api-access-gmwp8\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372612 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372627 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372641 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c06501d6-4873-4e09-a1c5-0620ba89da6d-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372655 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372670 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c06501d6-4873-4e09-a1c5-0620ba89da6d-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372684 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.372697 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.373025 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.374816 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.376859 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.385741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.378772 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.398280 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.397933 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.398280 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.398106 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.402771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.399865 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.402771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.401856 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "kube-api-access-v4wdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.406460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.405779 4918 generic.go:356] "Generic (PLEG): container finished" podID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerID="6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0" exitCode=0 Oct 30 22:02:25.406460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.405897 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" event={"ID":"b8244505-8b0c-4f00-aeca-abf790ed1d7d","Type":"ContainerDied","Data":"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0"} Oct 30 22:02:25.406460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.405954 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" event={"ID":"b8244505-8b0c-4f00-aeca-abf790ed1d7d","Type":"ContainerDied","Data":"59c5c2c66eee119562d2d052670bad507b1c1d75de30feb0d173b8df9747434b"} Oct 30 22:02:25.406460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.405983 4918 scope.go:122] "RemoveContainer" containerID="6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0" Oct 30 22:02:25.406460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.406110 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j" Oct 30 22:02:25.413887 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.408976 4918 generic.go:356] "Generic (PLEG): container finished" podID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerID="a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6" exitCode=0 Oct 30 22:02:25.413887 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.409081 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" event={"ID":"c06501d6-4873-4e09-a1c5-0620ba89da6d","Type":"ContainerDied","Data":"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6"} Oct 30 22:02:25.413887 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.409130 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" event={"ID":"c06501d6-4873-4e09-a1c5-0620ba89da6d","Type":"ContainerDied","Data":"0771bccfc1789bf3f5622fdd7fe300d0173e44c865b2738bf8065545176f13c6"} Oct 30 22:02:25.413887 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.409215 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4" Oct 30 22:02:25.414267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.414249 4918 generic.go:356] "Generic (PLEG): container finished" podID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerID="eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214" exitCode=0 Oct 30 22:02:25.414364 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.414347 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" event={"ID":"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec","Type":"ContainerDied","Data":"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214"} Oct 30 22:02:25.414464 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.414387 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" event={"ID":"df60cf13-cbc8-4742-b7d9-c0b9e8f38aec","Type":"ContainerDied","Data":"f15f14946f0f11574d7e479a5c382f3156d66b08f535f8b6cd9fdbd14dc88fb8"} Oct 30 22:02:25.414464 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.414459 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq" Oct 30 22:02:25.426678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.425720 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.426678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.425724 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.426678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.425744 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.426678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.425756 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.426678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.425773 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "kube-api-access-jzb4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.427096 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.426846 4918 scope.go:122] "RemoveContainer" containerID="6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0" Oct 30 22:02:25.427241 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.427213 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy" pod "b8244505-8b0c-4f00-aeca-abf790ed1d7d" (UID: "b8244505-8b0c-4f00-aeca-abf790ed1d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.427370 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:25.427248 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0\": container with ID starting with 6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0 not found: ID does not exist" containerID="6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0" Oct 30 22:02:25.427370 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.427279 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0"} err="failed to get container status \"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0\": rpc error: code = NotFound desc = could not find container \"6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0\": container with ID starting with 6e06cf8dcc27244764b39c5724e3fd63c0fb07164cdfac293323a41a6cbbcba0 not found: ID does not exist" Oct 30 22:02:25.427370 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.427302 4918 scope.go:122] "RemoveContainer" containerID="a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6" Oct 30 22:02:25.427636 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.427613 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token" pod "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" (UID: "df60cf13-cbc8-4742-b7d9-c0b9e8f38aec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.445267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.445184 4918 scope.go:122] "RemoveContainer" containerID="a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6" Oct 30 22:02:25.445561 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:25.445517 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6\": container with ID starting with a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6 not found: ID does not exist" containerID="a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6" Oct 30 22:02:25.445633 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.445550 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6"} err="failed to get container status \"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6\": rpc error: code = NotFound desc = could not find container \"a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6\": container with ID starting with a02ceff7340385a535ad455c315c05d88b387da1c0bd4844603546f28b14f9a6 not found: ID does not exist" Oct 30 22:02:25.445633 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.445575 4918 scope.go:122] "RemoveContainer" containerID="eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214" Oct 30 22:02:25.457342 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.457316 4918 scope.go:122] "RemoveContainer" containerID="eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214" Oct 30 22:02:25.457612 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:25.457590 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214\": container with ID starting with eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214 not found: ID does not exist" containerID="eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214" Oct 30 22:02:25.457688 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.457617 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214"} err="failed to get container status \"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214\": rpc error: code = NotFound desc = could not find container \"eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214\": container with ID starting with eda2147f4bb69b3efdd6d1a7c039b36d23200c0806a5a5d2b4a86a8b67465214 not found: ID does not exist" Oct 30 22:02:25.474005 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.473983 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474005 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474006 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474016 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474027 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzb4f\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-kube-api-access-jzb4f\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474037 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4wdg\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-kube-api-access-v4wdg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474046 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474054 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474062 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474070 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474078 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474085 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474093 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474102 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474110 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8244505-8b0c-4f00-aeca-abf790ed1d7d-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.474136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.474118 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.768231 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.768151 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:02:25.776211 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.776175 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-sw8j4"] Oct 30 22:02:25.796201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.796155 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.888320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.888270 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:02:25.893313 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.893281 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-gvnpq"] Oct 30 22:02:25.996617 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.996588 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:02:25.998942 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:25.998914 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.000076 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.000055 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-9895j"] Oct 30 22:02:26.103898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.103814 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 22:02:26.104131 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.104094 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" containerID="cri-o://a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70" gracePeriod=30 Oct 30 22:02:26.196967 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.196934 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.285629 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.285579 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" containerID="cri-o://02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318" gracePeriod=25 Oct 30 22:02:26.601861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.601815 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:02:26.602008 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:26.601880 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:02:27.431677 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:27.431642 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" path="/var/lib/kubelet/pods/b8244505-8b0c-4f00-aeca-abf790ed1d7d/volumes" Oct 30 22:02:27.432157 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:27.432142 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" path="/var/lib/kubelet/pods/c06501d6-4873-4e09-a1c5-0620ba89da6d/volumes" Oct 30 22:02:27.432596 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:27.432576 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" path="/var/lib/kubelet/pods/df60cf13-cbc8-4742-b7d9-c0b9e8f38aec/volumes" Oct 30 22:02:27.624851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:27.624820 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:29.157668 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.157635 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" containerID="cri-o://92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c" gracePeriod=45 Oct 30 22:02:29.424557 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.424480 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 22:02:29.448485 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.448451 4918 generic.go:356] "Generic (PLEG): container finished" podID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerID="92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c" exitCode=0 Oct 30 22:02:29.448600 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.448492 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" Oct 30 22:02:29.448600 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.448516 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" event={"ID":"4b01a127-e0e7-44c5-a3da-efe7971bfad1","Type":"ContainerDied","Data":"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c"} Oct 30 22:02:29.448600 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.448550 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl" event={"ID":"4b01a127-e0e7-44c5-a3da-efe7971bfad1","Type":"ContainerDied","Data":"85a643aff556b505d90e6109c5dcb6b24f7f9473e94f00e579d267ce4a615ae4"} Oct 30 22:02:29.448600 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.448571 4918 scope.go:122] "RemoveContainer" containerID="92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c" Oct 30 22:02:29.457590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.457560 4918 scope.go:122] "RemoveContainer" containerID="92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c" Oct 30 22:02:29.457890 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:29.457867 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c\": container with ID starting with 92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c not found: ID does not exist" containerID="92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c" Oct 30 22:02:29.457936 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.457898 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c"} err="failed to get container status \"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c\": rpc error: code = NotFound desc = could not find container \"92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c\": container with ID starting with 92c4e785bf67581645c7f6e40d646dce34a2a73aa00da26523e85322cb94fe2c not found: ID does not exist" Oct 30 22:02:29.524332 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524309 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524336 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524406 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\" (UniqueName: \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524452 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524542 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524529 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524571 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\") pod \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\" (UID: \"4b01a127-e0e7-44c5-a3da-efe7971bfad1\") " Oct 30 22:02:29.524811 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524768 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.524811 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524791 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.524984 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.524913 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.525192 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.525168 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.525307 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.525196 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.525307 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.525211 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.526534 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.526506 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.526620 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.526539 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:29.526845 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.526826 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng" pod "4b01a127-e0e7-44c5-a3da-efe7971bfad1" (UID: "4b01a127-e0e7-44c5-a3da-efe7971bfad1"). InnerVolumeSpecName "kube-api-access-q5cng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:29.625837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.625803 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b01a127-e0e7-44c5-a3da-efe7971bfad1-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.625837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.625833 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b01a127-e0e7-44c5-a3da-efe7971bfad1-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.625837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.625843 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cng\" (UniqueName: \"kubernetes.io/projected/4b01a127-e0e7-44c5-a3da-efe7971bfad1-kube-api-access-q5cng\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.874597 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.874562 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 22:02:29.880747 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:29.880716 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-991335/distinct-2-w5-991335-kserve-98fbdfccc-gtknl"] Oct 30 22:02:31.370904 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.370825 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 22:02:31.428944 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.428913 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" path="/var/lib/kubelet/pods/4b01a127-e0e7-44c5-a3da-efe7971bfad1/volumes" Oct 30 22:02:31.463597 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.463572 4918 generic.go:356] "Generic (PLEG): container finished" podID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerID="a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70" exitCode=0 Oct 30 22:02:31.463749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.463619 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" Oct 30 22:02:31.463749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.463665 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerDied","Data":"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70"} Oct 30 22:02:31.463749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.463728 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz" event={"ID":"ed5a2312-fa87-4a37-9202-2ab94864be0f","Type":"ContainerDied","Data":"99693ad7fbdfe089f6324741cb8897913d886232584e9aa588188dbc75e0aab9"} Oct 30 22:02:31.463860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.463760 4918 scope.go:122] "RemoveContainer" containerID="a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70" Oct 30 22:02:31.473451 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.473433 4918 scope.go:122] "RemoveContainer" containerID="a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2" Oct 30 22:02:31.482024 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.482010 4918 scope.go:122] "RemoveContainer" containerID="a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70" Oct 30 22:02:31.482278 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:31.482259 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70\": container with ID starting with a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70 not found: ID does not exist" containerID="a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70" Oct 30 22:02:31.482333 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.482292 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70"} err="failed to get container status \"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70\": rpc error: code = NotFound desc = could not find container \"a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70\": container with ID starting with a632dbe7deaa14ce95eeef8dfd33eb61b43c30ad065ed846fab40d72998b4b70 not found: ID does not exist" Oct 30 22:02:31.482333 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.482313 4918 scope.go:122] "RemoveContainer" containerID="a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2" Oct 30 22:02:31.482564 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:31.482545 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2\": container with ID starting with a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2 not found: ID does not exist" containerID="a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2" Oct 30 22:02:31.482616 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.482568 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2"} err="failed to get container status \"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2\": rpc error: code = NotFound desc = could not find container \"a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2\": container with ID starting with a6a44a4a825c54d1e59a318fcbeb1a44acfc00958a4e3e4d91efa573ea5038b2 not found: ID does not exist" Oct 30 22:02:31.543844 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.543821 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.543975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.543861 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.543975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.543897 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.543975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.543950 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544134 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.543985 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544134 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544068 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544134 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544107 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544173 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544217 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:31.544281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544229 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\") pod \"ed5a2312-fa87-4a37-9202-2ab94864be0f\" (UID: \"ed5a2312-fa87-4a37-9202-2ab94864be0f\") " Oct 30 22:02:31.544447 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544308 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:31.544634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544609 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:31.544728 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544681 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:31.544873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.544850 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:31.545349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.545325 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.545421 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.545355 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed5a2312-fa87-4a37-9202-2ab94864be0f-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.545421 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.545372 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.545421 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.545387 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.545421 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.545401 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.547257 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.547237 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "kube-api-access-pfrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:31.547257 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.547239 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:31.547381 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.547267 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:31.547452 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.547436 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo" pod "ed5a2312-fa87-4a37-9202-2ab94864be0f" (UID: "ed5a2312-fa87-4a37-9202-2ab94864be0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:31.638342 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.638265 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:31.646818 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.646793 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.646818 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.646819 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.647009 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.646833 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrml\" (UniqueName: \"kubernetes.io/projected/ed5a2312-fa87-4a37-9202-2ab94864be0f-kube-api-access-pfrml\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.647009 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.646847 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed5a2312-fa87-4a37-9202-2ab94864be0f-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:31.881652 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.881621 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 22:02:31.886096 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.886067 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq9xz"] Oct 30 22:02:31.931207 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931137 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:02:31.931737 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931718 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931738 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931756 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f21fca56-4a41-4e0a-880e-7380c80c2219" containerName="maas-api" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931763 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21fca56-4a41-4e0a-880e-7380c80c2219" containerName="maas-api" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931780 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931788 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931817 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" Oct 30 22:02:31.931850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931824 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931852 4918 cpu_manager.go:414] "containerMap: removing container" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931860 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931875 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.931883 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932024 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b01a127-e0e7-44c5-a3da-efe7971bfad1" containerName="main" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932039 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932051 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="df60cf13-cbc8-4742-b7d9-c0b9e8f38aec" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932061 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8244505-8b0c-4f00-aeca-abf790ed1d7d" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932074 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06501d6-4873-4e09-a1c5-0620ba89da6d" containerName="istio-proxy" Oct 30 22:02:31.932179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.932082 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21fca56-4a41-4e0a-880e-7380c80c2219" containerName="maas-api" Oct 30 22:02:31.937204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.937185 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:31.987934 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.987908 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-87eb6a-serving-cert\"" Oct 30 22:02:31.993183 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:31.993157 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:02:32.049807 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.049776 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.049975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.049824 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.050049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.050001 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwqz\" (UniqueName: \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.050119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.050089 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.151013 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.150986 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.151155 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.151028 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.151155 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.151085 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwqz\" (UniqueName: \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.151155 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.151136 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.151337 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:32.151195 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-serving-cert: secret "maas-api-e2e-api-b-87eb6a-serving-cert" not found Oct 30 22:02:32.151337 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:32.151260 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls podName:a4642900-9b51-4c2e-b205-88aeac96c24e nodeName:}" failed. No retries permitted until 2026-10-30 22:02:32.651246442 +0000 UTC m=+2933.344728660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls") pod "maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" (UID: "a4642900-9b51-4c2e-b205-88aeac96c24e") : secret "maas-api-e2e-api-b-87eb6a-serving-cert" not found Oct 30 22:02:32.151650 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.151629 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.153613 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.153592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.170562 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.169367 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.212166 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.212099 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwqz\" (UniqueName: \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.370993 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.370943 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.555729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.555629 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.656126 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.656093 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.658642 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.658617 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") pod \"maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.851377 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.851349 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:32.932152 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:32.932124 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 22:02:33.060473 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.060438 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\") pod \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " Oct 30 22:02:33.060627 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.060520 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\" (UniqueName: \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\") pod \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\" (UID: \"164c817d-3e33-4d6f-b000-6ea2d1f487a8\") " Oct 30 22:02:33.060863 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.060834 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume" pod "164c817d-3e33-4d6f-b000-6ea2d1f487a8" (UID: "164c817d-3e33-4d6f-b000-6ea2d1f487a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:33.062608 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.062588 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m" pod "164c817d-3e33-4d6f-b000-6ea2d1f487a8" (UID: "164c817d-3e33-4d6f-b000-6ea2d1f487a8"). InnerVolumeSpecName "kube-api-access-zn56m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:33.162331 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.162306 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164c817d-3e33-4d6f-b000-6ea2d1f487a8-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.162457 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.162337 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn56m\" (UniqueName: \"kubernetes.io/projected/164c817d-3e33-4d6f-b000-6ea2d1f487a8-kube-api-access-zn56m\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.281059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.281031 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:02:33.282163 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:02:33.282138 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4642900_9b51_4c2e_b205_88aeac96c24e.slice/crio-084ccf5a9742a58b2fb4502537c381a0739d7933b40e0aac769e90266996c2b8 WatchSource:0}: Error finding container 084ccf5a9742a58b2fb4502537c381a0739d7933b40e0aac769e90266996c2b8: Status 404 returned error can't find the container with id 084ccf5a9742a58b2fb4502537c381a0739d7933b40e0aac769e90266996c2b8 Oct 30 22:02:33.283721 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.283692 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:02:33.429783 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.429691 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" path="/var/lib/kubelet/pods/ed5a2312-fa87-4a37-9202-2ab94864be0f/volumes" Oct 30 22:02:33.478393 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.478365 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" event={"ID":"a4642900-9b51-4c2e-b205-88aeac96c24e","Type":"ContainerStarted","Data":"084ccf5a9742a58b2fb4502537c381a0739d7933b40e0aac769e90266996c2b8"} Oct 30 22:02:33.479680 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.479656 4918 generic.go:356] "Generic (PLEG): container finished" podID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerID="e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3" exitCode=0 Oct 30 22:02:33.479798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.479694 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" Oct 30 22:02:33.479798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.479749 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" event={"ID":"164c817d-3e33-4d6f-b000-6ea2d1f487a8","Type":"ContainerDied","Data":"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3"} Oct 30 22:02:33.479798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.479786 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b" event={"ID":"164c817d-3e33-4d6f-b000-6ea2d1f487a8","Type":"ContainerDied","Data":"b617e2323d17ab0a4e72ace879c62766478acd3de611fd6438993cdeb38f2c67"} Oct 30 22:02:33.479934 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.479808 4918 scope.go:122] "RemoveContainer" containerID="e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3" Oct 30 22:02:33.489231 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.489215 4918 scope.go:122] "RemoveContainer" containerID="e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3" Oct 30 22:02:33.489496 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:33.489475 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3\": container with ID starting with e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3 not found: ID does not exist" containerID="e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3" Oct 30 22:02:33.489559 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.489508 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3"} err="failed to get container status \"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3\": rpc error: code = NotFound desc = could not find container \"e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3\": container with ID starting with e88e159c2aef5b6ec8bbb267146375a2dd8db2987cbd201c6327d64a5e7077d3 not found: ID does not exist" Oct 30 22:02:33.641256 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.641229 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 22:02:33.650325 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:33.650301 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a2dced-6886c54799-xcr2b"] Oct 30 22:02:34.488409 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:34.488320 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" event={"ID":"a4642900-9b51-4c2e-b205-88aeac96c24e","Type":"ContainerStarted","Data":"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8"} Oct 30 22:02:34.488830 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:34.488427 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:34.603635 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:34.603579 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" podStartSLOduration=2.699497721 podStartE2EDuration="3.603565469s" totalImagesPullingTime="904.067748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:34.600921348 +0000 UTC m=+2935.294403584" watchObservedRunningTime="2026-10-30 22:02:34.603565469 +0000 UTC m=+2935.297047704" Oct 30 22:02:34.660716 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:34.660669 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:35.428779 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:35.428734 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" path="/var/lib/kubelet/pods/164c817d-3e33-4d6f-b000-6ea2d1f487a8/volumes" Oct 30 22:02:37.335219 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:37.335183 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:38.575625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:38.575591 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:38.765694 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:38.765662 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:39.690138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:39.690106 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.501873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:40.501838 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:02:41.222351 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.222315 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 22:02:41.222855 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.222545 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" podUID="26b7f92c-07d9-4272-a240-9966ac95e81f" containerName="maas-api" containerID="cri-o://afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588" gracePeriod=30 Oct 30 22:02:41.373522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.373480 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:41.484734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.484665 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 22:02:41.539743 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.539694 4918 generic.go:356] "Generic (PLEG): container finished" podID="26b7f92c-07d9-4272-a240-9966ac95e81f" containerID="afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588" exitCode=0 Oct 30 22:02:41.539881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.539747 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" Oct 30 22:02:41.539881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.539784 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" event={"ID":"26b7f92c-07d9-4272-a240-9966ac95e81f","Type":"ContainerDied","Data":"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588"} Oct 30 22:02:41.539881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.539850 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng" event={"ID":"26b7f92c-07d9-4272-a240-9966ac95e81f","Type":"ContainerDied","Data":"b6e960ff331128eeae39fbd1d9ef84502bd6f8670ace7d9de45884a181796763"} Oct 30 22:02:41.540036 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.539881 4918 scope.go:122] "RemoveContainer" containerID="afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588" Oct 30 22:02:41.547178 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547155 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\") pod \"26b7f92c-07d9-4272-a240-9966ac95e81f\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " Oct 30 22:02:41.547274 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547255 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\" (UniqueName: \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\") pod \"26b7f92c-07d9-4272-a240-9966ac95e81f\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " Oct 30 22:02:41.547329 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547303 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\") pod \"26b7f92c-07d9-4272-a240-9966ac95e81f\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " Oct 30 22:02:41.547414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547397 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") pod \"26b7f92c-07d9-4272-a240-9966ac95e81f\" (UID: \"26b7f92c-07d9-4272-a240-9966ac95e81f\") " Oct 30 22:02:41.547723 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547687 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca" pod "26b7f92c-07d9-4272-a240-9966ac95e81f" (UID: "26b7f92c-07d9-4272-a240-9966ac95e81f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:41.547891 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.547873 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b7f92c-07d9-4272-a240-9966ac95e81f-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.549333 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.549312 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs" pod "26b7f92c-07d9-4272-a240-9966ac95e81f" (UID: "26b7f92c-07d9-4272-a240-9966ac95e81f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:41.549509 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.549488 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls" pod "26b7f92c-07d9-4272-a240-9966ac95e81f" (UID: "26b7f92c-07d9-4272-a240-9966ac95e81f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:41.549816 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.549791 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5" pod "26b7f92c-07d9-4272-a240-9966ac95e81f" (UID: "26b7f92c-07d9-4272-a240-9966ac95e81f"). InnerVolumeSpecName "kube-api-access-lf4g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:41.550609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.550589 4918 scope.go:122] "RemoveContainer" containerID="afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588" Oct 30 22:02:41.550957 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:41.550934 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588\": container with ID starting with afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588 not found: ID does not exist" containerID="afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588" Oct 30 22:02:41.551037 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.550962 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588"} err="failed to get container status \"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588\": rpc error: code = NotFound desc = could not find container \"afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588\": container with ID starting with afb9ab9ddf9569571d7dfb2b10d3cb88deec88e00d044503ff4040e375481588 not found: ID does not exist" Oct 30 22:02:41.648693 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.648662 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.648693 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.648689 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4g5\" (UniqueName: \"kubernetes.io/projected/26b7f92c-07d9-4272-a240-9966ac95e81f-kube-api-access-lf4g5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.648877 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.648700 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26b7f92c-07d9-4272-a240-9966ac95e81f-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.958813 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.958775 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 22:02:41.961782 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:41.961754 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a2dced-d9f65dd4d-42kng"] Oct 30 22:02:42.295389 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:42.295359 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:42.481782 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:42.481743 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:42.797248 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:42.797214 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.046042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:43.046009 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.238374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:43.238343 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.428020 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:43.427990 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b7f92c-07d9-4272-a240-9966ac95e81f" path="/var/lib/kubelet/pods/26b7f92c-07d9-4272-a240-9966ac95e81f/volumes" Oct 30 22:02:43.778856 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:43.778829 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:44.660181 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:44.660136 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:45.082807 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.082727 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:02:45.083442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083408 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:45.083442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083432 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:45.083442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083443 4918 cpu_manager.go:414] "containerMap: removing container" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" Oct 30 22:02:45.083442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083449 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" Oct 30 22:02:45.083665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083477 4918 cpu_manager.go:414] "containerMap: removing container" podUID="26b7f92c-07d9-4272-a240-9966ac95e81f" containerName="maas-api" Oct 30 22:02:45.083665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083485 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b7f92c-07d9-4272-a240-9966ac95e81f" containerName="maas-api" Oct 30 22:02:45.083665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083574 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b7f92c-07d9-4272-a240-9966ac95e81f" containerName="maas-api" Oct 30 22:02:45.083665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083587 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="164c817d-3e33-4d6f-b000-6ea2d1f487a8" containerName="payload-pre-processing" Oct 30 22:02:45.083665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.083598 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a2312-fa87-4a37-9202-2ab94864be0f" containerName="istio-proxy" Oct 30 22:02:45.088550 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.088530 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.138082 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.138059 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:02:45.138201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.138132 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vmmk5\"" Oct 30 22:02:45.138201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.138133 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:02:45.184200 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.184174 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:02:45.186112 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186081 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186125 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186157 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186212 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186428 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186276 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186542 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186477 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.186542 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.186533 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mn6\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.188204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.188176 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:02:45.287664 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287636 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.287832 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287692 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.287832 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287751 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mn6\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.288047 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287856 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.288047 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287884 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.288047 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287913 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.288047 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.287967 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.288956 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.288923 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.290517 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.290485 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.290750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.290702 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.290879 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.290861 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.291082 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.291065 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.347607 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.347548 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mn6\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.351286 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.351265 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lv5jq\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.402591 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.402564 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:45.803634 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:45.803587 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:02:45.827903 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.827881 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:02:45.832763 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:02:45.832012 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce87cab_c966_4954_ae27_fcd6ae971c10.slice/crio-ab6cf0b9d46cfaf02720b2909a36f86505f2731e2655b11ef13e9cd3bb691927 WatchSource:0}: Error finding container ab6cf0b9d46cfaf02720b2909a36f86505f2731e2655b11ef13e9cd3bb691927: Status 404 returned error can't find the container with id ab6cf0b9d46cfaf02720b2909a36f86505f2731e2655b11ef13e9cd3bb691927 Oct 30 22:02:45.834188 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.834156 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:45.834273 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:45.834219 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:46.285496 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.285441 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.492214 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.492110 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.554157 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.554094 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 22:02:46.572915 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.572880 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" event={"ID":"3ce87cab-c966-4954-ae27-fcd6ae971c10","Type":"ContainerStarted","Data":"31355975a3a6de2c97a7354f2a8418b56fe6a1f5c3b1ff8715c155cc965c4001"} Oct 30 22:02:46.573088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.572950 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:46.573088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.572972 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" event={"ID":"3ce87cab-c966-4954-ae27-fcd6ae971c10","Type":"ContainerStarted","Data":"ab6cf0b9d46cfaf02720b2909a36f86505f2731e2655b11ef13e9cd3bb691927"} Oct 30 22:02:46.574496 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.574473 4918 generic.go:356] "Generic (PLEG): container finished" podID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerID="02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318" exitCode=0 Oct 30 22:02:46.574613 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.574516 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" Oct 30 22:02:46.574613 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.574588 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" event={"ID":"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b","Type":"ContainerDied","Data":"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318"} Oct 30 22:02:46.574757 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.574629 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw" event={"ID":"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b","Type":"ContainerDied","Data":"c27e234ca9b3f2bb44b32dc9e358dc8967d329945bbf96031b2d95c4cb07b0c5"} Oct 30 22:02:46.574757 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.574666 4918 scope.go:122] "RemoveContainer" containerID="02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318" Oct 30 22:02:46.575273 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.575250 4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lv5jq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:02:46.575390 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.575302 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:02:46.584530 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.584509 4918 scope.go:122] "RemoveContainer" containerID="02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318" Oct 30 22:02:46.584837 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:02:46.584816 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318\": container with ID starting with 02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318 not found: ID does not exist" containerID="02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318" Oct 30 22:02:46.584919 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.584845 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318"} err="failed to get container status \"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318\": rpc error: code = NotFound desc = could not find container \"02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318\": container with ID starting with 02dcecc000a52a5d7b994c155182c03d816118117f9ecaabe9c6370d44d73318 not found: ID does not exist" Oct 30 22:02:46.602317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.602286 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\") pod \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " Oct 30 22:02:46.602506 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.602488 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\" (UniqueName: \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\") pod \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\" (UID: \"b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b\") " Oct 30 22:02:46.602690 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.602670 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume" pod "b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" (UID: "b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:46.603207 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.603053 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.604777 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.604753 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v" pod "b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" (UID: "b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b"). InnerVolumeSpecName "kube-api-access-ctt2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:46.704325 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.704290 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctt2v\" (UniqueName: \"kubernetes.io/projected/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b-kube-api-access-ctt2v\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.831492 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:46.831387 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" podStartSLOduration=1.8313719800000001 podStartE2EDuration="1.83137198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:46.828138698 +0000 UTC m=+2947.521620935" watchObservedRunningTime="2026-10-30 22:02:46.83137198 +0000 UTC m=+2947.524854216" Oct 30 22:02:47.005508 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:47.005436 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 22:02:47.009960 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:47.009913 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-991335-699bc7685b-h5nsw"] Oct 30 22:02:47.428263 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:47.428227 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" path="/var/lib/kubelet/pods/b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b/volumes" Oct 30 22:02:47.583824 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:47.583796 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:02:47.923420 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:47.923382 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.667064 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:51.667009 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.907427 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:51.907387 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:52.106315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:52.106232 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:52.893394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:52.893351 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.075480 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:54.075440 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.278417 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:54.278375 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.333516 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:55.333482 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:57.667004 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:02:57.666967 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.512418 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:01.512378 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:02.240359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:02.240328 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:02.500254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:02.500177 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:03.922810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:03.922748 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.692701 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:05.692665 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.880643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:05.880612 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:06.561092 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.561047 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 22:03:06.561292 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.561248 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" podUID="94a2d290-6451-4544-a1e0-be827433e06f" containerName="maas-api" containerID="cri-o://46febd0b44c6344fb00cdcad360fc8ae3696bf1da65067edb6f917b1dfc4513c" gracePeriod=30 Oct 30 22:03:06.733614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.733578 4918 generic.go:356] "Generic (PLEG): container finished" podID="94a2d290-6451-4544-a1e0-be827433e06f" containerID="46febd0b44c6344fb00cdcad360fc8ae3696bf1da65067edb6f917b1dfc4513c" exitCode=0 Oct 30 22:03:06.734144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.733639 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" event={"ID":"94a2d290-6451-4544-a1e0-be827433e06f","Type":"ContainerDied","Data":"46febd0b44c6344fb00cdcad360fc8ae3696bf1da65067edb6f917b1dfc4513c"} Oct 30 22:03:06.779373 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.776733 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:06.847361 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.847337 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 22:03:06.929300 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.929269 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\") pod \"94a2d290-6451-4544-a1e0-be827433e06f\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " Oct 30 22:03:06.929444 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.929339 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\") pod \"94a2d290-6451-4544-a1e0-be827433e06f\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " Oct 30 22:03:06.929493 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.929446 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\") pod \"94a2d290-6451-4544-a1e0-be827433e06f\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " Oct 30 22:03:06.929553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.929521 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") pod \"94a2d290-6451-4544-a1e0-be827433e06f\" (UID: \"94a2d290-6451-4544-a1e0-be827433e06f\") " Oct 30 22:03:06.929889 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.929859 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca" pod "94a2d290-6451-4544-a1e0-be827433e06f" (UID: "94a2d290-6451-4544-a1e0-be827433e06f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:06.930119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.930093 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a2d290-6451-4544-a1e0-be827433e06f-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:06.931523 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.931501 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs" pod "94a2d290-6451-4544-a1e0-be827433e06f" (UID: "94a2d290-6451-4544-a1e0-be827433e06f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:06.931678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.931655 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls" pod "94a2d290-6451-4544-a1e0-be827433e06f" (UID: "94a2d290-6451-4544-a1e0-be827433e06f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:06.931890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:06.931868 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh" pod "94a2d290-6451-4544-a1e0-be827433e06f" (UID: "94a2d290-6451-4544-a1e0-be827433e06f"). InnerVolumeSpecName "kube-api-access-xztnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:07.031763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.031700 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.031763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.031759 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztnh\" (UniqueName: \"kubernetes.io/projected/94a2d290-6451-4544-a1e0-be827433e06f-kube-api-access-xztnh\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.031763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.031770 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a2d290-6451-4544-a1e0-be827433e06f-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.741265 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.741194 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" Oct 30 22:03:07.741265 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.741218 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8" event={"ID":"94a2d290-6451-4544-a1e0-be827433e06f","Type":"ContainerDied","Data":"f0e35f5b27172cf078d734b103395bf4abb136d1cdf49f3b43a7357ba2ac737c"} Oct 30 22:03:07.741694 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.741287 4918 scope.go:122] "RemoveContainer" containerID="46febd0b44c6344fb00cdcad360fc8ae3696bf1da65067edb6f917b1dfc4513c" Oct 30 22:03:07.898130 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.898096 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 22:03:07.945074 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:07.945047 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-991335-659d9d6cdb-qpbs8"] Oct 30 22:03:08.284238 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:08.284201 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.533102 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:08.533072 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.716689 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:08.716654 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.243619 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:09.243586 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.429337 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:09.429304 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a2d290-6451-4544-a1e0-be827433e06f" path="/var/lib/kubelet/pods/94a2d290-6451-4544-a1e0-be827433e06f/volumes" Oct 30 22:03:13.526528 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:13.526474 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:15.804619 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:03:15.804583 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:03:15.929996 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:15.929954 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.154117 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:16.154090 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.362049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:16.362015 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:17.459310 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:17.459258 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:17.641969 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:17.641929 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.213381 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:18.213346 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:22.914365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:22.914326 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.625998 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:27.625962 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.331988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:32.331950 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.934137 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:36.934100 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.914883 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.914852 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:03:39.928777 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.928738 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:03:39.931125 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.931094 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:03:39.934554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.934525 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:03:39.945641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.945623 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:03:39.950983 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:39.950966 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:03:41.527144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:41.527109 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.806301 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:03:45.806266 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:03:46.169538 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:46.169496 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.653685 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:46.653656 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.027232 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:49.027197 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.733187 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:50.733153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.118506 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:55.118464 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:56.631009 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:56.630971 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:56.903577 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:56.903488 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:57.100591 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:57.100559 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:57.614403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:03:57.614372 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:00.389362 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:00.389328 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:01.615608 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:01.615567 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.281226 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:04.281188 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.468590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:04.468556 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.669964 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:04.669932 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:05.579811 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:05.579776 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:06.964574 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:06.964536 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:12.084439 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:12.084403 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 22:04:12.084904 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:12.084613 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" containerID="cri-o://f70fc98ea244d819178e2587b9d3c86473c891269c507ab1d5eb4f7707ca9e10" gracePeriod=30 Oct 30 22:04:12.166545 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:12.166483 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:15.806049 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:15.806008 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:04:17.143340 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.143306 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:17.222245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.222205 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 22:04:17.242632 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.242606 4918 generic.go:356] "Generic (PLEG): container finished" podID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerID="f70fc98ea244d819178e2587b9d3c86473c891269c507ab1d5eb4f7707ca9e10" exitCode=0 Oct 30 22:04:17.242786 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.242685 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" event={"ID":"dcb15ad2-f865-4b84-b4ca-30b3925a134c","Type":"ContainerDied","Data":"f70fc98ea244d819178e2587b9d3c86473c891269c507ab1d5eb4f7707ca9e10"} Oct 30 22:04:17.345896 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.345864 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 22:04:17.348063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.348041 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 22:04:17.450414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450381 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450460 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450501 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450529 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450558 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450882 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450617 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450882 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450652 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450882 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450696 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.450882 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.450790 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\") pod \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\" (UID: \"dcb15ad2-f865-4b84-b4ca-30b3925a134c\") " Oct 30 22:04:17.451355 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.451321 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.451496 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.451420 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.451837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.451794 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.452293 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.452265 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.452407 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.452355 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:17.453687 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.453659 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "kube-api-access-7v2v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:17.453852 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.453742 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:17.454088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.454058 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:17.454307 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.454287 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy" pod "dcb15ad2-f865-4b84-b4ca-30b3925a134c" (UID: "dcb15ad2-f865-4b84-b4ca-30b3925a134c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:17.552674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552640 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2v7\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-kube-api-access-7v2v7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552672 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552686 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-envoy\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552700 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-data\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552733 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552747 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-credential-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552756 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istiod-ca-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552765 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb15ad2-f865-4b84-b4ca-30b3925a134c-istio-podinfo\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.552929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.552786 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb15ad2-f865-4b84-b4ca-30b3925a134c-workload-socket\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.659922 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.659885 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:17.851992 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:17.851909 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:18.250348 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:18.250321 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" Oct 30 22:04:18.250824 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:18.250360 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m" event={"ID":"dcb15ad2-f865-4b84-b4ca-30b3925a134c","Type":"ContainerDied","Data":"0bc849af7767de3fed4d3c0693928fd60a9feaf7e183965e1ac21e1e491938ea"} Oct 30 22:04:18.250824 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:18.250424 4918 scope.go:122] "RemoveContainer" containerID="f70fc98ea244d819178e2587b9d3c86473c891269c507ab1d5eb4f7707ca9e10" Oct 30 22:04:18.407097 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:18.407067 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 22:04:18.409828 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:18.409805 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-v9v6m"] Oct 30 22:04:19.428367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:19.428325 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" path="/var/lib/kubelet/pods/dcb15ad2-f865-4b84-b4ca-30b3925a134c/volumes" Oct 30 22:04:19.731094 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:19.731006 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:04:19.841998 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:19.841963 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:04:20.148439 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:20.148390 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:20.339843 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:20.339813 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:22.039048 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:22.039009 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:22.223511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:22.223471 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" containerID="cri-o://8cc74bc6dd971b9cc28d3636dd045482a5e41cb454a98899d5ed10110bfa71c4" gracePeriod=25 Oct 30 22:04:22.346500 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:22.346430 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" containerID="cri-o://556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c" gracePeriod=25 Oct 30 22:04:22.531188 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:22.531116 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:22.718826 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:22.718794 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:24.731951 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:24.731909 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" podUID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" containerName="payload-processing" containerID="cri-o://239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b" gracePeriod=25 Oct 30 22:04:24.842435 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:24.842365 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" podUID="10e753ee-db7c-44a5-ac63-a9416dcd979a" containerName="payload-pre-processing" containerID="cri-o://968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e" gracePeriod=25 Oct 30 22:04:25.003420 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.003360 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:04:25.104504 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.104481 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:04:25.131201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.131168 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\") pod \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " Oct 30 22:04:25.131344 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.131254 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\" (UniqueName: \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\") pod \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\" (UID: \"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4\") " Oct 30 22:04:25.131612 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.131583 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume" pod "fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" (UID: "fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:25.131861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.131839 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.134204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.134170 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5" pod "fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" (UID: "fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4"). InnerVolumeSpecName "kube-api-access-fv6f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.232857 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.232828 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\") pod \"10e753ee-db7c-44a5-ac63-a9416dcd979a\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " Oct 30 22:04:25.233011 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.232938 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\" (UniqueName: \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\") pod \"10e753ee-db7c-44a5-ac63-a9416dcd979a\" (UID: \"10e753ee-db7c-44a5-ac63-a9416dcd979a\") " Oct 30 22:04:25.233211 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.233188 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume" pod "10e753ee-db7c-44a5-ac63-a9416dcd979a" (UID: "10e753ee-db7c-44a5-ac63-a9416dcd979a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:25.233356 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.233336 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e753ee-db7c-44a5-ac63-a9416dcd979a-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.233356 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.233356 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6f5\" (UniqueName: \"kubernetes.io/projected/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4-kube-api-access-fv6f5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.235144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.235116 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7" pod "10e753ee-db7c-44a5-ac63-a9416dcd979a" (UID: "10e753ee-db7c-44a5-ac63-a9416dcd979a"). InnerVolumeSpecName "kube-api-access-k7cb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.315869 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.315800 4918 generic.go:356] "Generic (PLEG): container finished" podID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" containerID="239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b" exitCode=0 Oct 30 22:04:25.315869 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.315855 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" Oct 30 22:04:25.316070 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.315887 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" event={"ID":"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4","Type":"ContainerDied","Data":"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b"} Oct 30 22:04:25.316070 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.315983 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf" event={"ID":"fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4","Type":"ContainerDied","Data":"83db4a7fa4f091307fd7b9654aa3788b13490c7e6f0c0b3fb47edd52163c2cd0"} Oct 30 22:04:25.316070 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.316015 4918 scope.go:122] "RemoveContainer" containerID="239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b" Oct 30 22:04:25.317412 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.317385 4918 generic.go:356] "Generic (PLEG): container finished" podID="10e753ee-db7c-44a5-ac63-a9416dcd979a" containerID="968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e" exitCode=0 Oct 30 22:04:25.317543 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.317425 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" Oct 30 22:04:25.317543 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.317445 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" event={"ID":"10e753ee-db7c-44a5-ac63-a9416dcd979a","Type":"ContainerDied","Data":"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e"} Oct 30 22:04:25.317543 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.317486 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj" event={"ID":"10e753ee-db7c-44a5-ac63-a9416dcd979a","Type":"ContainerDied","Data":"afb9d247c0ca8411b5470af76b44c084639ec3cf741157390201e8d7cfc34db8"} Oct 30 22:04:25.328045 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.328026 4918 scope.go:122] "RemoveContainer" containerID="239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b" Oct 30 22:04:25.328378 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:25.328350 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b\": container with ID starting with 239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b not found: ID does not exist" containerID="239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b" Oct 30 22:04:25.328475 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.328388 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b"} err="failed to get container status \"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b\": rpc error: code = NotFound desc = could not find container \"239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b\": container with ID starting with 239790b79bfdbc003f0214b94863cb984cc7ead8e4d433a6f40da1df25a8312b not found: ID does not exist" Oct 30 22:04:25.328475 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.328414 4918 scope.go:122] "RemoveContainer" containerID="968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e" Oct 30 22:04:25.334891 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.334863 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cb7\" (UniqueName: \"kubernetes.io/projected/10e753ee-db7c-44a5-ac63-a9416dcd979a-kube-api-access-k7cb7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.337254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.337238 4918 scope.go:122] "RemoveContainer" containerID="968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e" Oct 30 22:04:25.337526 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:25.337508 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e\": container with ID starting with 968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e not found: ID does not exist" containerID="968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e" Oct 30 22:04:25.337594 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.337533 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e"} err="failed to get container status \"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e\": rpc error: code = NotFound desc = could not find container \"968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e\": container with ID starting with 968721b96f2882c6399a4bcd680f1c51dac50996a08454f48a8f965ceacd6e4e not found: ID does not exist" Oct 30 22:04:25.474549 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.474514 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:04:25.477664 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.477636 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87eb6a-6bbcf9f946-p8szf"] Oct 30 22:04:25.588969 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.588878 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:04:25.593267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:25.593242 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87eb6a-7fbf858fff-pl8hj"] Oct 30 22:04:26.943779 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:26.943743 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:27.427955 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:27.427927 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e753ee-db7c-44a5-ac63-a9416dcd979a" path="/var/lib/kubelet/pods/10e753ee-db7c-44a5-ac63-a9416dcd979a/volumes" Oct 30 22:04:27.428329 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:27.428316 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" path="/var/lib/kubelet/pods/fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4/volumes" Oct 30 22:04:28.545061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:28.545024 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:28.545516 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:28.545024 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:30.207029 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:30.206993 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.423618 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:30.423586 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.250221 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:32.250184 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:36.445230 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.445198 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:04:36.445726 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.445351 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" podUID="a4642900-9b51-4c2e-b205-88aeac96c24e" containerName="maas-api" containerID="cri-o://84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8" gracePeriod=30 Oct 30 22:04:36.605302 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.605221 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:36.712138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.712069 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:04:36.845327 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845291 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\") pod \"a4642900-9b51-4c2e-b205-88aeac96c24e\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " Oct 30 22:04:36.845522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\") pod \"a4642900-9b51-4c2e-b205-88aeac96c24e\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " Oct 30 22:04:36.845522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845370 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") pod \"a4642900-9b51-4c2e-b205-88aeac96c24e\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " Oct 30 22:04:36.845655 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845532 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\" (UniqueName: \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\") pod \"a4642900-9b51-4c2e-b205-88aeac96c24e\" (UID: \"a4642900-9b51-4c2e-b205-88aeac96c24e\") " Oct 30 22:04:36.845732 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845647 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca" pod "a4642900-9b51-4c2e-b205-88aeac96c24e" (UID: "a4642900-9b51-4c2e-b205-88aeac96c24e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:36.846009 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.845987 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4642900-9b51-4c2e-b205-88aeac96c24e-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:36.847764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.847739 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls" pod "a4642900-9b51-4c2e-b205-88aeac96c24e" (UID: "a4642900-9b51-4c2e-b205-88aeac96c24e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:36.847966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.847944 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz" pod "a4642900-9b51-4c2e-b205-88aeac96c24e" (UID: "a4642900-9b51-4c2e-b205-88aeac96c24e"). InnerVolumeSpecName "kube-api-access-nkwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:36.848101 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.848083 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs" pod "a4642900-9b51-4c2e-b205-88aeac96c24e" (UID: "a4642900-9b51-4c2e-b205-88aeac96c24e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:36.946936 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.946908 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:36.947055 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.946940 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4642900-9b51-4c2e-b205-88aeac96c24e-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:36.947055 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:36.946956 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwqz\" (UniqueName: \"kubernetes.io/projected/a4642900-9b51-4c2e-b205-88aeac96c24e-kube-api-access-nkwqz\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.395802 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.395766 4918 generic.go:356] "Generic (PLEG): container finished" podID="a4642900-9b51-4c2e-b205-88aeac96c24e" containerID="84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8" exitCode=0 Oct 30 22:04:37.395978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.395809 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" Oct 30 22:04:37.395978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.395851 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" event={"ID":"a4642900-9b51-4c2e-b205-88aeac96c24e","Type":"ContainerDied","Data":"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8"} Oct 30 22:04:37.395978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.395905 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr" event={"ID":"a4642900-9b51-4c2e-b205-88aeac96c24e","Type":"ContainerDied","Data":"084ccf5a9742a58b2fb4502537c381a0739d7933b40e0aac769e90266996c2b8"} Oct 30 22:04:37.395978 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.395926 4918 scope.go:122] "RemoveContainer" containerID="84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8" Oct 30 22:04:37.406375 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.406356 4918 scope.go:122] "RemoveContainer" containerID="84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8" Oct 30 22:04:37.406632 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:37.406612 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8\": container with ID starting with 84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8 not found: ID does not exist" containerID="84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8" Oct 30 22:04:37.406744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.406645 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8"} err="failed to get container status \"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8\": rpc error: code = NotFound desc = could not find container \"84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8\": container with ID starting with 84e19fefda82c7a7ae15547164dcce5ccc7a52e40aded59b4ee3ebe2d7466ff8 not found: ID does not exist" Oct 30 22:04:37.560988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.560943 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:04:37.606531 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:37.606501 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87eb6a-59cb9fff8f-j97nr"] Oct 30 22:04:38.240020 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:38.239989 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:38.470909 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:38.470859 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:38.545008 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:38.544927 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:38.545169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:38.544926 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:38.671718 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:38.671677 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:39.226453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:39.226415 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:39.427692 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:39.427656 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4642900-9b51-4c2e-b205-88aeac96c24e" path="/var/lib/kubelet/pods/a4642900-9b51-4c2e-b205-88aeac96c24e/volumes" Oct 30 22:04:42.432511 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.432478 4918 generic.go:356] "Generic (PLEG): container finished" podID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerID="8cc74bc6dd971b9cc28d3636dd045482a5e41cb454a98899d5ed10110bfa71c4" exitCode=0 Oct 30 22:04:42.433007 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.432597 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" event={"ID":"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100","Type":"ContainerDied","Data":"8cc74bc6dd971b9cc28d3636dd045482a5e41cb454a98899d5ed10110bfa71c4"} Oct 30 22:04:42.516414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.516380 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 22:04:42.605086 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.605047 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\" (UniqueName: \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\") pod \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " Oct 30 22:04:42.605086 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.605093 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\") pod \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\" (UID: \"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100\") " Oct 30 22:04:42.605546 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.605522 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume" pod "5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" (UID: "5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:42.607331 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.607295 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd" pod "5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" (UID: "5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100"). InnerVolumeSpecName "kube-api-access-qmsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:42.627896 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.627855 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 22:04:42.705890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.705852 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\" (UniqueName: \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\") pod \"68d081d8-3123-4d0c-b73c-fa690d3b1156\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " Oct 30 22:04:42.706082 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.705991 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\") pod \"68d081d8-3123-4d0c-b73c-fa690d3b1156\" (UID: \"68d081d8-3123-4d0c-b73c-fa690d3b1156\") " Oct 30 22:04:42.706454 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.706428 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmsmd\" (UniqueName: \"kubernetes.io/projected/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-kube-api-access-qmsmd\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:42.706549 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.706460 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:42.706549 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.706499 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume" pod "68d081d8-3123-4d0c-b73c-fa690d3b1156" (UID: "68d081d8-3123-4d0c-b73c-fa690d3b1156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:42.708895 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.708866 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj" pod "68d081d8-3123-4d0c-b73c-fa690d3b1156" (UID: "68d081d8-3123-4d0c-b73c-fa690d3b1156"). InnerVolumeSpecName "kube-api-access-kf8hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:42.807179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.807094 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8hj\" (UniqueName: \"kubernetes.io/projected/68d081d8-3123-4d0c-b73c-fa690d3b1156-kube-api-access-kf8hj\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:42.807179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:42.807128 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d081d8-3123-4d0c-b73c-fa690d3b1156-plugins-config-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:43.447673 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.447642 4918 generic.go:356] "Generic (PLEG): container finished" podID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerID="556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c" exitCode=0 Oct 30 22:04:43.448159 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.447753 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" event={"ID":"68d081d8-3123-4d0c-b73c-fa690d3b1156","Type":"ContainerDied","Data":"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c"} Oct 30 22:04:43.448159 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.447801 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" event={"ID":"68d081d8-3123-4d0c-b73c-fa690d3b1156","Type":"ContainerDied","Data":"3e72b506509e3a95512eb8ddc2d3bd98d3df1c75495df90de70f333baaad7582"} Oct 30 22:04:43.448159 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.447852 4918 scope.go:122] "RemoveContainer" containerID="556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c" Oct 30 22:04:43.448159 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.448006 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5" Oct 30 22:04:43.450442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.450416 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" event={"ID":"5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100","Type":"ContainerDied","Data":"2b9b4910054034b89459ba313e65448e6dda77e62d0fd65920844fa04a3e2e17"} Oct 30 22:04:43.450580 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.450509 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw" Oct 30 22:04:43.460815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.460790 4918 scope.go:122] "RemoveContainer" containerID="556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c" Oct 30 22:04:43.461098 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:43.461079 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c\": container with ID starting with 556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c not found: ID does not exist" containerID="556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c" Oct 30 22:04:43.461179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.461105 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c"} err="failed to get container status \"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c\": rpc error: code = NotFound desc = could not find container \"556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c\": container with ID starting with 556375da11a388a81b2f5c13aed403b9b741a88989cbcf24345ac95f0345d33c not found: ID does not exist" Oct 30 22:04:43.461179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.461122 4918 scope.go:122] "RemoveContainer" containerID="8cc74bc6dd971b9cc28d3636dd045482a5e41cb454a98899d5ed10110bfa71c4" Oct 30 22:04:43.493261 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.493196 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:43.605555 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.605521 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 22:04:43.652379 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.652343 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ed76b8-5cdc648995-6v5d5"] Oct 30 22:04:43.723692 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.723663 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:43.755048 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.754982 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 22:04:43.758998 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.758975 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ed76b8-f477996db-gblhw"] Oct 30 22:04:43.929767 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:43.929693 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:44.233416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:44.233388 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:45.427942 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:45.427903 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" path="/var/lib/kubelet/pods/5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100/volumes" Oct 30 22:04:45.428407 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:45.428389 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" path="/var/lib/kubelet/pods/68d081d8-3123-4d0c-b73c-fa690d3b1156/volumes" Oct 30 22:04:45.802559 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:04:45.802481 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:04:46.173524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:46.173476 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:46.423441 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:46.423412 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:46.661613 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:46.661561 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.004346 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:47.004256 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.211966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:47.211928 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.451998 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:47.451967 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:48.547898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:48.547866 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:50.230627 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:50.230595 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:50.503818 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:50.503734 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:50.710379 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:50.710346 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:51.280685 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:51.280654 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.149975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:52.149933 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:56.118417 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:56.118382 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.252197 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:04:59.252163 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.607871 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:00.607832 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.992153 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:00.992121 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.802105 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:05:15.802064 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:05:20.655497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:20.655415 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 22:05:20.655908 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:20.655634 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" podUID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" containerName="maas-api" containerID="cri-o://56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389" gracePeriod=30 Oct 30 22:05:20.828815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:20.828785 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:20.922038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:20.921966 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 22:05:21.084815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.084771 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\") pod \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " Oct 30 22:05:21.085138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.085111 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\") pod \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " Oct 30 22:05:21.085426 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.085404 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\" (UniqueName: \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\") pod \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " Oct 30 22:05:21.085581 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.085565 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") pod \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\" (UID: \"cbad3c9f-1581-454f-9b7e-d3c6c7580300\") " Oct 30 22:05:21.091466 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.091350 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca" pod "cbad3c9f-1581-454f-9b7e-d3c6c7580300" (UID: "cbad3c9f-1581-454f-9b7e-d3c6c7580300"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:21.103945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.095913 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs" pod "cbad3c9f-1581-454f-9b7e-d3c6c7580300" (UID: "cbad3c9f-1581-454f-9b7e-d3c6c7580300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:21.103945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.096518 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls" pod "cbad3c9f-1581-454f-9b7e-d3c6c7580300" (UID: "cbad3c9f-1581-454f-9b7e-d3c6c7580300"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:21.103945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.103370 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv" pod "cbad3c9f-1581-454f-9b7e-d3c6c7580300" (UID: "cbad3c9f-1581-454f-9b7e-d3c6c7580300"). InnerVolumeSpecName "kube-api-access-gh2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:21.187224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.187137 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2gv\" (UniqueName: \"kubernetes.io/projected/cbad3c9f-1581-454f-9b7e-d3c6c7580300-kube-api-access-gh2gv\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:21.187224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.187168 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:21.187224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.187179 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbad3c9f-1581-454f-9b7e-d3c6c7580300-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:21.187224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.187188 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbad3c9f-1581-454f-9b7e-d3c6c7580300-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:21.686040 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.686008 4918 generic.go:356] "Generic (PLEG): container finished" podID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" containerID="56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389" exitCode=0 Oct 30 22:05:21.686541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.686076 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" Oct 30 22:05:21.686541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.686101 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" event={"ID":"cbad3c9f-1581-454f-9b7e-d3c6c7580300","Type":"ContainerDied","Data":"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389"} Oct 30 22:05:21.686541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.686172 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6" event={"ID":"cbad3c9f-1581-454f-9b7e-d3c6c7580300","Type":"ContainerDied","Data":"370a0e80650dddd69ed02136f8ace7a2201e69fa16cb6441f9357a07a83c2f18"} Oct 30 22:05:21.686541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.686208 4918 scope.go:122] "RemoveContainer" containerID="56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389" Oct 30 22:05:21.698652 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.698625 4918 scope.go:122] "RemoveContainer" containerID="56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389" Oct 30 22:05:21.698940 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:05:21.698922 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389\": container with ID starting with 56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389 not found: ID does not exist" containerID="56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389" Oct 30 22:05:21.698999 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.698950 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389"} err="failed to get container status \"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389\": rpc error: code = NotFound desc = could not find container \"56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389\": container with ID starting with 56ea4e87d4c0ac861c5851ba739d60335bbd896707994776a6164f5aeb3e2389 not found: ID does not exist" Oct 30 22:05:21.843604 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.843579 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 22:05:21.890784 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:21.890756 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-afbe5b-5f67cc7cc6-kjqc6"] Oct 30 22:05:22.363561 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:22.363522 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.668399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:22.668320 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:23.430002 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:23.429966 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" path="/var/lib/kubelet/pods/cbad3c9f-1581-454f-9b7e-d3c6c7580300/volumes" Oct 30 22:05:23.526536 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:23.526508 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.198297 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:29.198197 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.392894 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:29.392858 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:30.941880 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:30.941846 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.484688 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:31.484648 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.687851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:31.687801 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.889005 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:31.888972 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:37.126049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.126016 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:37.206630 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.206597 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 22:05:37.206868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.206835 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-744f599cc5-hxk6h" podUID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" containerName="manager" containerID="cri-o://64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf" gracePeriod=30 Oct 30 22:05:37.452303 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.452279 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 22:05:37.542026 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.541997 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\") pod \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " Oct 30 22:05:37.542026 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.542039 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\") pod \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " Oct 30 22:05:37.542266 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.542155 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\" (UniqueName: \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\") pod \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\" (UID: \"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c\") " Oct 30 22:05:37.544112 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.544083 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs" pod "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" (UID: "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:37.544227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.544200 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert" pod "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" (UID: "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:37.544272 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.544260 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9" pod "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" (UID: "1f1c8f68-3fb9-4a35-a187-1ef25cf0617c"). InnerVolumeSpecName "kube-api-access-69cx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:37.643940 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.643906 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cx9\" (UniqueName: \"kubernetes.io/projected/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-kube-api-access-69cx9\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.643940 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.643933 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.643940 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.643945 4918 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c-webhook-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.793938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.793853 4918 generic.go:356] "Generic (PLEG): container finished" podID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" containerID="64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf" exitCode=0 Oct 30 22:05:37.793938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.793897 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744f599cc5-hxk6h" Oct 30 22:05:37.794139 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.793936 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744f599cc5-hxk6h" event={"ID":"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c","Type":"ContainerDied","Data":"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf"} Oct 30 22:05:37.794139 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.793992 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744f599cc5-hxk6h" event={"ID":"1f1c8f68-3fb9-4a35-a187-1ef25cf0617c","Type":"ContainerDied","Data":"0c7435af9cedaa0cbd156dcbf966a61f62fcc071afcfff604388fa2b2a7d0ebe"} Oct 30 22:05:37.794139 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.794016 4918 scope.go:122] "RemoveContainer" containerID="64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf" Oct 30 22:05:37.806061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.806036 4918 scope.go:122] "RemoveContainer" containerID="64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf" Oct 30 22:05:37.806336 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:05:37.806313 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf\": container with ID starting with 64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf not found: ID does not exist" containerID="64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf" Oct 30 22:05:37.806401 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.806344 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf"} err="failed to get container status \"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf\": rpc error: code = NotFound desc = could not find container \"64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf\": container with ID starting with 64ca3922e50ee440804ca93ba9220d98acb74693df3c1b7fdd430d5e273670bf not found: ID does not exist" Oct 30 22:05:37.953499 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.953466 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 22:05:37.999240 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:37.999208 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-744f599cc5-hxk6h"] Oct 30 22:05:39.428114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:39.428084 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" path="/var/lib/kubelet/pods/1f1c8f68-3fb9-4a35-a187-1ef25cf0617c/volumes" Oct 30 22:05:45.802611 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:05:45.802580 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:05:47.386032 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:05:47.385998 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:14.626423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:06:14.626386 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:15.801597 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:06:15.801560 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:06:16.957674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:06:16.957641 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.803635 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:06:45.803596 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:06:47.702058 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:06:47.702025 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.530864 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:06:55.530825 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.744558 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:06:55.744523 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.801170 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:07:15.801124 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:07:25.674794 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:25.674755 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.868804 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:25.868773 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.325459 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:27.325413 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:31.711730 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:31.710789 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qzk86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:07:31.711730 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:31.710874 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qzk86" podUID="b85f2df0-2e73-44aa-9f4f-caf9bec10c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:45.801562 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:07:45.801526 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:07:49.630665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:49.630632 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:49.819849 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:07:49.819816 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.812660 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:08:15.812608 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:08:17.636873 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:17.636828 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:08:17.637380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:17.637081 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerName="discovery" containerID="cri-o://31355975a3a6de2c97a7354f2a8418b56fe6a1f5c3b1ff8715c155cc965c4001" gracePeriod=30 Oct 30 22:08:17.807803 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:17.807762 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerID="31355975a3a6de2c97a7354f2a8418b56fe6a1f5c3b1ff8715c155cc965c4001" exitCode=0 Oct 30 22:08:17.808028 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:17.807965 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" event={"ID":"3ce87cab-c966-4954-ae27-fcd6ae971c10","Type":"ContainerDied","Data":"31355975a3a6de2c97a7354f2a8418b56fe6a1f5c3b1ff8715c155cc965c4001"} Oct 30 22:08:17.915546 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:17.915516 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:08:18.012720 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012682 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.012938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012765 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.012938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012821 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.012938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012865 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.012938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012884 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.012938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012906 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.013253 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.012969 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\") pod \"3ce87cab-c966-4954-ae27-fcd6ae971c10\" (UID: \"3ce87cab-c966-4954-ae27-fcd6ae971c10\") " Oct 30 22:08:18.013846 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.013805 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:18.015341 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015307 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "kube-api-access-r4mn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.015470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015425 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.015637 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015609 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.015751 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015684 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.015751 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015696 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.015939 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.015919 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs" pod "3ce87cab-c966-4954-ae27-fcd6ae971c10" (UID: "3ce87cab-c966-4954-ae27-fcd6ae971c10"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:18.114512 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114462 4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-kubeconfig\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114512 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114505 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mn6\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-kube-api-access-r4mn6\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114512 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114521 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-dns-cert\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114533 4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce87cab-c966-4954-ae27-fcd6ae971c10-local-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114545 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-token\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114559 4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ce87cab-c966-4954-ae27-fcd6ae971c10-cacerts\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.114831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.114571 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce87cab-c966-4954-ae27-fcd6ae971c10-istio-csr-ca-configmap\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.816720 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.816679 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" Oct 30 22:08:18.817229 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.816726 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq" event={"ID":"3ce87cab-c966-4954-ae27-fcd6ae971c10","Type":"ContainerDied","Data":"ab6cf0b9d46cfaf02720b2909a36f86505f2731e2655b11ef13e9cd3bb691927"} Oct 30 22:08:18.817229 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:18.816792 4918 scope.go:122] "RemoveContainer" containerID="31355975a3a6de2c97a7354f2a8418b56fe6a1f5c3b1ff8715c155cc965c4001" Oct 30 22:08:19.022100 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:19.022060 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:08:19.069093 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:19.069001 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lv5jq"] Oct 30 22:08:19.428101 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:19.428065 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" path="/var/lib/kubelet/pods/3ce87cab-c966-4954-ae27-fcd6ae971c10/volumes" Oct 30 22:08:33.138958 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:33.138921 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:33.330796 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:33.330763 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:34.143810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.143774 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d"] Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144288 4918 cpu_manager.go:414] "containerMap: removing container" podUID="10e753ee-db7c-44a5-ac63-a9416dcd979a" containerName="payload-pre-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144301 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e753ee-db7c-44a5-ac63-a9416dcd979a" containerName="payload-pre-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144322 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a4642900-9b51-4c2e-b205-88aeac96c24e" containerName="maas-api" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144331 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4642900-9b51-4c2e-b205-88aeac96c24e" containerName="maas-api" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144348 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144354 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144366 4918 cpu_manager.go:414] "containerMap: removing container" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144371 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144380 4918 cpu_manager.go:414] "containerMap: removing container" podUID="94a2d290-6451-4544-a1e0-be827433e06f" containerName="maas-api" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144385 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a2d290-6451-4544-a1e0-be827433e06f" containerName="maas-api" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144398 4918 cpu_manager.go:414] "containerMap: removing container" podUID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" containerName="payload-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144405 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" containerName="payload-processing" Oct 30 22:08:34.144403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144415 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerName="discovery" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144420 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerName="discovery" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144429 4918 cpu_manager.go:414] "containerMap: removing container" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144433 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144448 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" containerName="maas-api" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144454 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" containerName="maas-api" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144461 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" containerName="manager" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144465 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" containerName="manager" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144473 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144480 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144546 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce87cab-c966-4954-ae27-fcd6ae971c10" containerName="discovery" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144559 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb15ad2-f865-4b84-b4ca-30b3925a134c" containerName="istio-proxy" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144570 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbad3c9f-1581-454f-9b7e-d3c6c7580300" containerName="maas-api" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144578 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1c8f68-3fb9-4a35-a187-1ef25cf0617c" containerName="manager" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144585 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc1cf7c-1aa4-4cca-92bf-7aae2b33a100" containerName="payload-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144590 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e753ee-db7c-44a5-ac63-a9416dcd979a" containerName="payload-pre-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144597 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4642900-9b51-4c2e-b205-88aeac96c24e" containerName="maas-api" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144603 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ec3b7-9a1c-4e86-92f6-4a2f6df0274b" containerName="payload-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144610 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d081d8-3123-4d0c-b73c-fa690d3b1156" containerName="payload-pre-processing" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144615 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a2d290-6451-4544-a1e0-be827433e06f" containerName="maas-api" Oct 30 22:08:34.145114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.144622 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe56b5a6-df99-4a59-a2fe-f2ef5dcc0bd4" containerName="payload-processing" Oct 30 22:08:34.149307 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.149279 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.248349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.248306 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:34.248349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.248342 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:08:34.251375 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.251336 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d"] Oct 30 22:08:34.280334 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.280291 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.280526 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.280457 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.280569 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.280547 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnmm\" (UniqueName: \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.382182 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.382136 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.382388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.382214 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnmm\" (UniqueName: \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.382388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.382258 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.382989 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.382966 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.384811 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.384791 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.440750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.440643 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnmm\" (UniqueName: \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\") pod \"maas-api-revoke-sub-046e1343-zdz4d\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.464608 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.464566 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:34.839837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.839807 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d"] Oct 30 22:08:34.841876 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:08:34.841839 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95e6b0c_f27e_44bc_97ba_ee17a0c8fd94.slice/crio-c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17 WatchSource:0}: Error finding container c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17: Status 404 returned error can't find the container with id c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17 Oct 30 22:08:34.919701 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:34.919664 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" event={"ID":"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94","Type":"ContainerStarted","Data":"c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17"} Oct 30 22:08:35.927641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:35.927603 4918 generic.go:356] "Generic (PLEG): container finished" podID="b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" containerID="fd085651bdbb5ab8c4ac7068e254c897d51a9b813c2032f10cedc41c586487c9" exitCode=0 Oct 30 22:08:35.928087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:35.927690 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" event={"ID":"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94","Type":"ContainerDied","Data":"fd085651bdbb5ab8c4ac7068e254c897d51a9b813c2032f10cedc41c586487c9"} Oct 30 22:08:37.083054 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.083010 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:37.210945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.210842 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\") pod \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " Oct 30 22:08:37.211140 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.210973 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\") pod \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " Oct 30 22:08:37.211140 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.211074 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\" (UniqueName: \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\") pod \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\" (UID: \"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94\") " Oct 30 22:08:37.211435 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.211389 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca" pod "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" (UID: "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:37.213277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.213239 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca" pod "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" (UID: "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:37.213781 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.213733 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm" pod "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" (UID: "b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94"). InnerVolumeSpecName "kube-api-access-wmnmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:37.312218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.312169 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.312218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.312212 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnmm\" (UniqueName: \"kubernetes.io/projected/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-kube-api-access-wmnmm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.312218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.312228 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.943380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.943339 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" event={"ID":"b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94","Type":"ContainerDied","Data":"c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17"} Oct 30 22:08:37.943380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.943384 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d" Oct 30 22:08:37.943610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:37.943404 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33ab173d3816ee126a9888a98ca8bd4843440568726854ad9c4f54cff9cff17" Oct 30 22:08:39.959609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.959574 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:08:39.973586 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.973548 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:08:39.977846 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.977817 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:08:39.978660 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.978638 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:08:39.991065 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.991035 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:08:39.996056 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:39.996030 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:08:44.306883 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.306849 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.943868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.943832 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l"] Oct 30 22:08:44.944402 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.944381 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" containerName="revoke-keys" Oct 30 22:08:44.944402 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.944399 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" containerName="revoke-keys" Oct 30 22:08:44.944610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.944480 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" containerName="revoke-keys" Oct 30 22:08:44.948129 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:44.948103 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.000043 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.000001 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:08:45.000043 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.000005 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:45.049819 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.049769 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l"] Oct 30 22:08:45.102568 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.102522 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9rj\" (UniqueName: \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.102837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.102604 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.102837 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.102737 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.203643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.203551 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9rj\" (UniqueName: \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.203643 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.203630 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.203890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.203652 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.204380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.204356 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.206092 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.206072 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.262121 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.262084 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9rj\" (UniqueName: \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\") pod \"maas-api-revoke-sub-d3fba056-x5k6l\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.264851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.264830 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:45.638918 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.638888 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l"] Oct 30 22:08:45.640470 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:08:45.640439 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857e6222_7e5b_4372_be1b_b162487fc04a.slice/crio-dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299 WatchSource:0}: Error finding container dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299: Status 404 returned error can't find the container with id dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299 Oct 30 22:08:45.802879 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:08:45.802828 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:08:45.997745 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.997683 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" event={"ID":"857e6222-7e5b-4372-be1b-b162487fc04a","Type":"ContainerStarted","Data":"080251e4841e94724bc41704722af996eed653a122e831e1fea6a8d428da66db"} Oct 30 22:08:45.997955 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:45.997775 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" event={"ID":"857e6222-7e5b-4372-be1b-b162487fc04a","Type":"ContainerStarted","Data":"dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299"} Oct 30 22:08:46.108135 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:46.108073 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" podStartSLOduration=2.108054366 podStartE2EDuration="2.108054366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:46.107504767 +0000 UTC m=+3306.800987004" watchObservedRunningTime="2026-10-30 22:08:46.108054366 +0000 UTC m=+3306.801536604" Oct 30 22:08:47.005832 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:47.005698 4918 generic.go:356] "Generic (PLEG): container finished" podID="857e6222-7e5b-4372-be1b-b162487fc04a" containerID="080251e4841e94724bc41704722af996eed653a122e831e1fea6a8d428da66db" exitCode=0 Oct 30 22:08:47.005832 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:47.005735 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" event={"ID":"857e6222-7e5b-4372-be1b-b162487fc04a","Type":"ContainerDied","Data":"080251e4841e94724bc41704722af996eed653a122e831e1fea6a8d428da66db"} Oct 30 22:08:48.204578 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.204538 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:48.336821 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.336719 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\" (UniqueName: \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\") pod \"857e6222-7e5b-4372-be1b-b162487fc04a\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " Oct 30 22:08:48.337078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.337053 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\") pod \"857e6222-7e5b-4372-be1b-b162487fc04a\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " Oct 30 22:08:48.337364 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.337283 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\") pod \"857e6222-7e5b-4372-be1b-b162487fc04a\" (UID: \"857e6222-7e5b-4372-be1b-b162487fc04a\") " Oct 30 22:08:48.337520 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.337488 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca" pod "857e6222-7e5b-4372-be1b-b162487fc04a" (UID: "857e6222-7e5b-4372-be1b-b162487fc04a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:48.337991 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.337958 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:48.340629 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.340599 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca" pod "857e6222-7e5b-4372-be1b-b162487fc04a" (UID: "857e6222-7e5b-4372-be1b-b162487fc04a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:48.340913 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.340886 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj" pod "857e6222-7e5b-4372-be1b-b162487fc04a" (UID: "857e6222-7e5b-4372-be1b-b162487fc04a"). InnerVolumeSpecName "kube-api-access-hf9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:48.377163 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.377120 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:48.439087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.439035 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857e6222-7e5b-4372-be1b-b162487fc04a-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:48.439087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:48.439083 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9rj\" (UniqueName: \"kubernetes.io/projected/857e6222-7e5b-4372-be1b-b162487fc04a-kube-api-access-hf9rj\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:49.045418 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:49.045374 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" event={"ID":"857e6222-7e5b-4372-be1b-b162487fc04a","Type":"ContainerDied","Data":"dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299"} Oct 30 22:08:49.045875 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:49.045845 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l" Oct 30 22:08:49.046169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:49.045862 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd29a031fc4db17ae29f906fa3f5a410f85cb5d43c794873009160aa6dc93299" Oct 30 22:08:55.117336 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:08:55.117291 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:02.318463 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:02.318418 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.806374 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:09:15.806331 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:09:19.923087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:19.923053 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h"] Oct 30 22:09:19.923733 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:19.923695 4918 cpu_manager.go:414] "containerMap: removing container" podUID="857e6222-7e5b-4372-be1b-b162487fc04a" containerName="revoke-keys" Oct 30 22:09:19.923851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:19.923736 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857e6222-7e5b-4372-be1b-b162487fc04a" containerName="revoke-keys" Oct 30 22:09:19.923851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:19.923846 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="857e6222-7e5b-4372-be1b-b162487fc04a" containerName="revoke-keys" Oct 30 22:09:19.927294 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:19.927274 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.003190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.003155 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.003313 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.003211 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.003374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.003320 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg85d\" (UniqueName: \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.024515 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.024488 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:09:20.024515 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.024496 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:20.027317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.027297 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h"] Oct 30 22:09:20.104033 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.104010 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg85d\" (UniqueName: \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.104124 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.104058 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.104124 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.104091 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.104654 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.104638 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.106513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.106490 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.162530 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.162499 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg85d\" (UniqueName: \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\") pod \"maas-api-revoke-sub-ae594979-p9z6h\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.242987 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.242931 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:20.610322 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:20.610298 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h"] Oct 30 22:09:20.611914 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:09:20.611887 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741ff9d6_feb0_4a26_ba42_cfc36413a61e.slice/crio-8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e WatchSource:0}: Error finding container 8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e: Status 404 returned error can't find the container with id 8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e Oct 30 22:09:21.249067 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:21.248979 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" event={"ID":"741ff9d6-feb0-4a26-ba42-cfc36413a61e","Type":"ContainerStarted","Data":"9b77bcc2c3d816681fa0b3c1053a646f9fb1750dc343cba628fc6a99e158bc35"} Oct 30 22:09:21.249430 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:21.249065 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" event={"ID":"741ff9d6-feb0-4a26-ba42-cfc36413a61e","Type":"ContainerStarted","Data":"8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e"} Oct 30 22:09:21.361477 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:21.361429 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" podStartSLOduration=2.361415938 podStartE2EDuration="2.361415938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:21.358180855 +0000 UTC m=+3342.051663092" watchObservedRunningTime="2026-10-30 22:09:21.361415938 +0000 UTC m=+3342.054898173" Oct 30 22:09:22.256218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:22.256185 4918 generic.go:356] "Generic (PLEG): container finished" podID="741ff9d6-feb0-4a26-ba42-cfc36413a61e" containerID="9b77bcc2c3d816681fa0b3c1053a646f9fb1750dc343cba628fc6a99e158bc35" exitCode=0 Oct 30 22:09:22.256604 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:22.256268 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" event={"ID":"741ff9d6-feb0-4a26-ba42-cfc36413a61e","Type":"ContainerDied","Data":"9b77bcc2c3d816681fa0b3c1053a646f9fb1750dc343cba628fc6a99e158bc35"} Oct 30 22:09:23.405667 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.405640 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:23.536095 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.536018 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\") pod \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " Oct 30 22:09:23.536244 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.536201 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\" (UniqueName: \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\") pod \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " Oct 30 22:09:23.536320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.536262 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\") pod \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\" (UID: \"741ff9d6-feb0-4a26-ba42-cfc36413a61e\") " Oct 30 22:09:23.536444 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.536351 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca" pod "741ff9d6-feb0-4a26-ba42-cfc36413a61e" (UID: "741ff9d6-feb0-4a26-ba42-cfc36413a61e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:23.536647 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.536624 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.538359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.538335 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d" pod "741ff9d6-feb0-4a26-ba42-cfc36413a61e" (UID: "741ff9d6-feb0-4a26-ba42-cfc36413a61e"). InnerVolumeSpecName "kube-api-access-bg85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:23.538471 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.538338 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca" pod "741ff9d6-feb0-4a26-ba42-cfc36413a61e" (UID: "741ff9d6-feb0-4a26-ba42-cfc36413a61e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:23.637250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.637220 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg85d\" (UniqueName: \"kubernetes.io/projected/741ff9d6-feb0-4a26-ba42-cfc36413a61e-kube-api-access-bg85d\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.637250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:23.637243 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ff9d6-feb0-4a26-ba42-cfc36413a61e-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:24.269416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:24.269388 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" Oct 30 22:09:24.269591 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:24.269427 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h" event={"ID":"741ff9d6-feb0-4a26-ba42-cfc36413a61e","Type":"ContainerDied","Data":"8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e"} Oct 30 22:09:24.269591 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:24.269480 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8207390d7039c617cfabcc480808750221ce8f4f765fb84dfaad95d60c11f05e" Oct 30 22:09:35.335767 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:35.335731 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.526702 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:35.526671 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:45.802097 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:09:45.802052 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:09:48.040782 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.040748 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s"] Oct 30 22:09:48.041302 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.041237 4918 cpu_manager.go:414] "containerMap: removing container" podUID="741ff9d6-feb0-4a26-ba42-cfc36413a61e" containerName="revoke-keys" Oct 30 22:09:48.041302 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.041250 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741ff9d6-feb0-4a26-ba42-cfc36413a61e" containerName="revoke-keys" Oct 30 22:09:48.041380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.041329 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="741ff9d6-feb0-4a26-ba42-cfc36413a61e" containerName="revoke-keys" Oct 30 22:09:48.044639 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.044616 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.103433 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.103397 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdprm\" (UniqueName: \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.103583 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.103526 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.103583 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.103567 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.142644 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.142623 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:48.142783 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.142679 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:09:48.145636 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.145613 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s"] Oct 30 22:09:48.204950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.204927 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdprm\" (UniqueName: \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.205061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.204985 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.205061 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.205013 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.205686 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.205669 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.207300 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.207280 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.262807 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.262778 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdprm\" (UniqueName: \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\") pod \"maas-api-revoke-sub-a4aef7d7-fzz8s\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.358041 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.357990 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:48.726359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.726325 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s"] Oct 30 22:09:48.730607 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:09:48.730575 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb98266_69db_4f7f_b0e7_1d44e0bb6179.slice/crio-7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27 WatchSource:0}: Error finding container 7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27: Status 404 returned error can't find the container with id 7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27 Oct 30 22:09:48.787923 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:48.787897 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:49.429277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:49.429235 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" event={"ID":"9bb98266-69db-4f7f-b0e7-1d44e0bb6179","Type":"ContainerStarted","Data":"e41c0df01141dd2789c75584a0821c4ba26c6ca0283574eb39733168585e299b"} Oct 30 22:09:49.429661 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:49.429298 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" event={"ID":"9bb98266-69db-4f7f-b0e7-1d44e0bb6179","Type":"ContainerStarted","Data":"7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27"} Oct 30 22:09:49.539045 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:49.538990 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" podStartSLOduration=2.5389744309999998 podStartE2EDuration="2.538974431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:49.537895527 +0000 UTC m=+3370.231377764" watchObservedRunningTime="2026-10-30 22:09:49.538974431 +0000 UTC m=+3370.232456668" Oct 30 22:09:50.436939 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:50.436898 4918 generic.go:356] "Generic (PLEG): container finished" podID="9bb98266-69db-4f7f-b0e7-1d44e0bb6179" containerID="e41c0df01141dd2789c75584a0821c4ba26c6ca0283574eb39733168585e299b" exitCode=0 Oct 30 22:09:50.437385 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:50.437013 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" event={"ID":"9bb98266-69db-4f7f-b0e7-1d44e0bb6179","Type":"ContainerDied","Data":"e41c0df01141dd2789c75584a0821c4ba26c6ca0283574eb39733168585e299b"} Oct 30 22:09:51.593217 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.593194 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:51.638996 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.638970 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\") pod \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " Oct 30 22:09:51.639112 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.639034 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\") pod \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " Oct 30 22:09:51.639165 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.639126 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\" (UniqueName: \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\") pod \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\" (UID: \"9bb98266-69db-4f7f-b0e7-1d44e0bb6179\") " Oct 30 22:09:51.639359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.639330 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca" pod "9bb98266-69db-4f7f-b0e7-1d44e0bb6179" (UID: "9bb98266-69db-4f7f-b0e7-1d44e0bb6179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:51.641055 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.641028 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca" pod "9bb98266-69db-4f7f-b0e7-1d44e0bb6179" (UID: "9bb98266-69db-4f7f-b0e7-1d44e0bb6179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:51.641139 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.641107 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm" pod "9bb98266-69db-4f7f-b0e7-1d44e0bb6179" (UID: "9bb98266-69db-4f7f-b0e7-1d44e0bb6179"). InnerVolumeSpecName "kube-api-access-bdprm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:51.740497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.740441 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdprm\" (UniqueName: \"kubernetes.io/projected/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-kube-api-access-bdprm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.740497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.740464 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.740497 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:51.740474 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb98266-69db-4f7f-b0e7-1d44e0bb6179-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:52.450234 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:52.450194 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" Oct 30 22:09:52.450234 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:52.450226 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s" event={"ID":"9bb98266-69db-4f7f-b0e7-1d44e0bb6179","Type":"ContainerDied","Data":"7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27"} Oct 30 22:09:52.450467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:09:52.450274 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff51094fc72da315a034b7d2146c762510bf61747e6a70890f5a3ada74dbe27" Oct 30 22:10:14.558175 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:14.558143 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.801895 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:10:15.801858 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:10:18.830205 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.830169 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd"] Oct 30 22:10:18.830661 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.830648 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9bb98266-69db-4f7f-b0e7-1d44e0bb6179" containerName="revoke-keys" Oct 30 22:10:18.830661 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.830662 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb98266-69db-4f7f-b0e7-1d44e0bb6179" containerName="revoke-keys" Oct 30 22:10:18.830771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.830761 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb98266-69db-4f7f-b0e7-1d44e0bb6179" containerName="revoke-keys" Oct 30 22:10:18.833997 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.833981 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:18.885670 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.885646 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:18.932715 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.932671 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:10:18.935579 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:18.935559 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd"] Oct 30 22:10:19.007270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.007247 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.007394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.007355 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997wv\" (UniqueName: \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.007394 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.007380 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.108095 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.108035 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997wv\" (UniqueName: \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.108095 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.108070 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.108276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.108147 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.108796 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.108776 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.110334 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.110315 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.176541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.176514 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997wv\" (UniqueName: \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\") pod \"maas-api-revoke-sub-79805ca9-7tshd\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.447091 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.447067 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:19.824040 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:19.824012 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd"] Oct 30 22:10:19.824957 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:10:19.824926 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814e49e3_6e6f_4985_b595_e2f9b37ee0ac.slice/crio-927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a WatchSource:0}: Error finding container 927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a: Status 404 returned error can't find the container with id 927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a Oct 30 22:10:20.627892 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:20.627859 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" event={"ID":"814e49e3-6e6f-4985-b595-e2f9b37ee0ac","Type":"ContainerStarted","Data":"07dd74e43a3d099a96111ca3d8f9d43ea1e3096336b5236b34d019104f8bb914"} Oct 30 22:10:20.628267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:20.627909 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" event={"ID":"814e49e3-6e6f-4985-b595-e2f9b37ee0ac","Type":"ContainerStarted","Data":"927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a"} Oct 30 22:10:20.740335 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:20.740291 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" podStartSLOduration=2.740277637 podStartE2EDuration="2.740277637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:20.737847976 +0000 UTC m=+3401.431330214" watchObservedRunningTime="2026-10-30 22:10:20.740277637 +0000 UTC m=+3401.433759872" Oct 30 22:10:21.635471 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:21.635431 4918 generic.go:356] "Generic (PLEG): container finished" podID="814e49e3-6e6f-4985-b595-e2f9b37ee0ac" containerID="07dd74e43a3d099a96111ca3d8f9d43ea1e3096336b5236b34d019104f8bb914" exitCode=0 Oct 30 22:10:21.635966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:21.635517 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" event={"ID":"814e49e3-6e6f-4985-b595-e2f9b37ee0ac","Type":"ContainerDied","Data":"07dd74e43a3d099a96111ca3d8f9d43ea1e3096336b5236b34d019104f8bb914"} Oct 30 22:10:22.779191 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.779161 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:22.840054 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.840025 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\") pod \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " Oct 30 22:10:22.840180 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.840069 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\" (UniqueName: \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\") pod \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " Oct 30 22:10:22.840332 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.840310 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca" pod "814e49e3-6e6f-4985-b595-e2f9b37ee0ac" (UID: "814e49e3-6e6f-4985-b595-e2f9b37ee0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:22.841985 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.841963 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv" pod "814e49e3-6e6f-4985-b595-e2f9b37ee0ac" (UID: "814e49e3-6e6f-4985-b595-e2f9b37ee0ac"). InnerVolumeSpecName "kube-api-access-997wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:22.940907 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.940852 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\") pod \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\" (UID: \"814e49e3-6e6f-4985-b595-e2f9b37ee0ac\") " Oct 30 22:10:22.941133 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.941119 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:22.941178 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.941135 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997wv\" (UniqueName: \"kubernetes.io/projected/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-kube-api-access-997wv\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:22.942748 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:22.942724 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca" pod "814e49e3-6e6f-4985-b595-e2f9b37ee0ac" (UID: "814e49e3-6e6f-4985-b595-e2f9b37ee0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:23.042066 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.042041 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814e49e3-6e6f-4985-b595-e2f9b37ee0ac-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:23.648480 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.648451 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" Oct 30 22:10:23.648641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.648485 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd" event={"ID":"814e49e3-6e6f-4985-b595-e2f9b37ee0ac","Type":"ContainerDied","Data":"927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a"} Oct 30 22:10:23.648641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.648536 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927e85a6eab9715da1f8fd46be432af230d7e14acd43035c1d78e521920d6d0a" Oct 30 22:10:23.676899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.676868 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:23.894583 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:23.894549 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:42.737436 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.737398 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq"] Oct 30 22:10:42.737922 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.737905 4918 cpu_manager.go:414] "containerMap: removing container" podUID="814e49e3-6e6f-4985-b595-e2f9b37ee0ac" containerName="revoke-keys" Oct 30 22:10:42.737969 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.737922 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814e49e3-6e6f-4985-b595-e2f9b37ee0ac" containerName="revoke-keys" Oct 30 22:10:42.738040 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.738030 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="814e49e3-6e6f-4985-b595-e2f9b37ee0ac" containerName="revoke-keys" Oct 30 22:10:42.741220 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.741199 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.792842 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.792754 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:42.819526 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.819488 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.819678 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.819569 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzsq\" (UniqueName: \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.819782 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.819741 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.839774 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.839753 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:10:42.842604 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.842587 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq"] Oct 30 22:10:42.920779 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.920755 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsq\" (UniqueName: \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.920871 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.920842 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.920916 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.920873 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.921456 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.921438 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.923206 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.923184 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:42.980149 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:42.980129 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzsq\" (UniqueName: \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\") pod \"maas-api-revoke-sub-0b7e8edb-ql4cq\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:43.054525 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:43.054474 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:43.422805 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:10:43.422760 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282d0cec_79e3_4d2b_8d24_70e0930d7e8a.slice/crio-46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5 WatchSource:0}: Error finding container 46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5: Status 404 returned error can't find the container with id 46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5 Oct 30 22:10:43.428550 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:43.428527 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq"] Oct 30 22:10:43.793465 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:43.793433 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" event={"ID":"282d0cec-79e3-4d2b-8d24-70e0930d7e8a","Type":"ContainerStarted","Data":"12c3f16724b09d190b7a903778d96bd22245b889e36c0a359c52c561ec5cf59a"} Oct 30 22:10:43.793874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:43.793483 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" event={"ID":"282d0cec-79e3-4d2b-8d24-70e0930d7e8a","Type":"ContainerStarted","Data":"46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5"} Oct 30 22:10:43.908619 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:43.908574 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" podStartSLOduration=1.908561707 podStartE2EDuration="1.908561707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:43.905725626 +0000 UTC m=+3424.599207855" watchObservedRunningTime="2026-10-30 22:10:43.908561707 +0000 UTC m=+3424.602043942" Oct 30 22:10:44.801966 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:44.801932 4918 generic.go:356] "Generic (PLEG): container finished" podID="282d0cec-79e3-4d2b-8d24-70e0930d7e8a" containerID="12c3f16724b09d190b7a903778d96bd22245b889e36c0a359c52c561ec5cf59a" exitCode=0 Oct 30 22:10:44.802362 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:44.801984 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" event={"ID":"282d0cec-79e3-4d2b-8d24-70e0930d7e8a","Type":"ContainerDied","Data":"12c3f16724b09d190b7a903778d96bd22245b889e36c0a359c52c561ec5cf59a"} Oct 30 22:10:45.802945 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:10:45.802907 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:10:45.956353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:45.956329 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:46.050826 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.050797 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\") pod \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " Oct 30 22:10:46.050954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.050873 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\") pod \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " Oct 30 22:10:46.050954 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.050924 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\" (UniqueName: \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\") pod \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\" (UID: \"282d0cec-79e3-4d2b-8d24-70e0930d7e8a\") " Oct 30 22:10:46.051196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.051170 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca" pod "282d0cec-79e3-4d2b-8d24-70e0930d7e8a" (UID: "282d0cec-79e3-4d2b-8d24-70e0930d7e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:46.052862 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.052840 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca" pod "282d0cec-79e3-4d2b-8d24-70e0930d7e8a" (UID: "282d0cec-79e3-4d2b-8d24-70e0930d7e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:46.053005 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.052960 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq" pod "282d0cec-79e3-4d2b-8d24-70e0930d7e8a" (UID: "282d0cec-79e3-4d2b-8d24-70e0930d7e8a"). InnerVolumeSpecName "kube-api-access-zlzsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:46.151933 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.151903 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:46.151933 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.151934 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzsq\" (UniqueName: \"kubernetes.io/projected/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-kube-api-access-zlzsq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:46.152118 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.151950 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282d0cec-79e3-4d2b-8d24-70e0930d7e8a-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:46.815625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.815584 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" event={"ID":"282d0cec-79e3-4d2b-8d24-70e0930d7e8a","Type":"ContainerDied","Data":"46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5"} Oct 30 22:10:46.815625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.815611 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq" Oct 30 22:10:46.816081 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:46.815639 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46316806485e9e4b76fb1b28c42fe852f0006709ef21360cc2f2aa7efc5eddb5" Oct 30 22:10:49.079476 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:49.079444 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:49.365870 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:49.365836 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.136749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.136715 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq"] Oct 30 22:10:54.137204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.137179 4918 cpu_manager.go:414] "containerMap: removing container" podUID="282d0cec-79e3-4d2b-8d24-70e0930d7e8a" containerName="revoke-keys" Oct 30 22:10:54.137204 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.137193 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282d0cec-79e3-4d2b-8d24-70e0930d7e8a" containerName="revoke-keys" Oct 30 22:10:54.137304 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.137292 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="282d0cec-79e3-4d2b-8d24-70e0930d7e8a" containerName="revoke-keys" Oct 30 22:10:54.140834 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.140813 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.192031 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.192003 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:54.196399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.196372 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq"] Oct 30 22:10:54.222371 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.222339 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.222523 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.222392 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4j7\" (UniqueName: \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.222574 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.222528 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.238646 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.238621 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:10:54.323416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.323390 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.323539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.323435 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4j7\" (UniqueName: \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.323539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.323470 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.324099 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.324075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.325785 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.325765 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.381461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.381435 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4j7\" (UniqueName: \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\") pod \"maas-api-revoke-sub-3d4156f1-ht8bq\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.455682 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.455630 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:54.833989 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.833959 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq"] Oct 30 22:10:54.835801 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:10:54.835774 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccb0c49_0e1e_4ea1_b58e_a90fd1c28d89.slice/crio-f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0 WatchSource:0}: Error finding container f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0: Status 404 returned error can't find the container with id f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0 Oct 30 22:10:54.860764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:54.860740 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" event={"ID":"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89","Type":"ContainerStarted","Data":"f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0"} Oct 30 22:10:55.868810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:55.868778 4918 generic.go:356] "Generic (PLEG): container finished" podID="7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" containerID="659f3e2208646ceaf17a3bd223542834464e5bc4cf71995712cd4e91e12b7ec2" exitCode=0 Oct 30 22:10:55.869193 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:55.868855 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" event={"ID":"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89","Type":"ContainerDied","Data":"659f3e2208646ceaf17a3bd223542834464e5bc4cf71995712cd4e91e12b7ec2"} Oct 30 22:10:57.277810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.277773 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:57.366090 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.366052 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\") pod \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " Oct 30 22:10:57.366277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.366137 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\") pod \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " Oct 30 22:10:57.366277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.366195 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\" (UniqueName: \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\") pod \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\" (UID: \"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89\") " Oct 30 22:10:57.366590 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.366562 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca" pod "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" (UID: "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:57.368258 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.368235 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca" pod "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" (UID: "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:57.368340 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.368304 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7" pod "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" (UID: "7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89"). InnerVolumeSpecName "kube-api-access-ql4j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:57.467116 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.467082 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:57.467249 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.467121 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:57.467249 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.467139 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4j7\" (UniqueName: \"kubernetes.io/projected/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89-kube-api-access-ql4j7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:57.886961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.886929 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" Oct 30 22:10:57.886961 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.886934 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq" event={"ID":"7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89","Type":"ContainerDied","Data":"f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0"} Oct 30 22:10:57.887183 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:10:57.886993 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f12aaec83d69f052ba1007ca3e46aea8195beb6570958168c4ee7ffa575124c0" Oct 30 22:11:07.438210 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:07.438158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:15.815761 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:11:15.815700 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:11:28.023972 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.023886 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq"] Oct 30 22:11:28.074638 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.074607 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ht8bq"] Oct 30 22:11:28.149657 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.149628 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m"] Oct 30 22:11:28.150136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.150122 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" containerName="revoke-keys" Oct 30 22:11:28.150182 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.150137 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" containerName="revoke-keys" Oct 30 22:11:28.150224 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.150216 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" containerName="revoke-keys" Oct 30 22:11:28.153397 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.153380 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.206697 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.206674 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:28.206927 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.206907 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:11:28.209072 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.209045 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m"] Oct 30 22:11:28.224899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.224869 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nq4\" (UniqueName: \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.224997 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.224920 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.225087 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.225063 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.326013 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.325927 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nq4\" (UniqueName: \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.326013 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.325972 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.326278 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.326024 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.326838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.326814 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.328440 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.328421 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.388085 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.388065 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nq4\" (UniqueName: \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\") pod \"maas-api-revoke-sub-3d4156f1-gjm2m\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.466078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.466048 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:28.840513 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:11:28.840479 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea0b743_2b8f_416c_a115_e5598bba97f7.slice/crio-99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d WatchSource:0}: Error finding container 99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d: Status 404 returned error can't find the container with id 99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d Oct 30 22:11:28.843540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:28.843519 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m"] Oct 30 22:11:29.117480 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:29.117452 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" event={"ID":"3ea0b743-2b8f-416c-a115-e5598bba97f7","Type":"ContainerStarted","Data":"24c7f4c39432352419aa76d699805910a9cb522d3737a684016f3ca9f91f81ae"} Oct 30 22:11:29.117806 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:29.117509 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" event={"ID":"3ea0b743-2b8f-416c-a115-e5598bba97f7","Type":"ContainerStarted","Data":"99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d"} Oct 30 22:11:29.235095 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:29.235051 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" podStartSLOduration=1.235038708 podStartE2EDuration="1.235038708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.234763533 +0000 UTC m=+3469.928245769" watchObservedRunningTime="2026-10-30 22:11:29.235038708 +0000 UTC m=+3469.928520943" Oct 30 22:11:29.427391 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:29.427331 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89" path="/var/lib/kubelet/pods/7ccb0c49-0e1e-4ea1-b58e-a90fd1c28d89/volumes" Oct 30 22:11:30.125020 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:30.124983 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ea0b743-2b8f-416c-a115-e5598bba97f7" containerID="24c7f4c39432352419aa76d699805910a9cb522d3737a684016f3ca9f91f81ae" exitCode=0 Oct 30 22:11:30.125461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:30.125022 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" event={"ID":"3ea0b743-2b8f-416c-a115-e5598bba97f7","Type":"ContainerDied","Data":"24c7f4c39432352419aa76d699805910a9cb522d3737a684016f3ca9f91f81ae"} Oct 30 22:11:31.281268 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.281244 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:31.352983 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.352958 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\") pod \"3ea0b743-2b8f-416c-a115-e5598bba97f7\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " Oct 30 22:11:31.353088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.353038 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\" (UniqueName: \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\") pod \"3ea0b743-2b8f-416c-a115-e5598bba97f7\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " Oct 30 22:11:31.353088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.353084 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\") pod \"3ea0b743-2b8f-416c-a115-e5598bba97f7\" (UID: \"3ea0b743-2b8f-416c-a115-e5598bba97f7\") " Oct 30 22:11:31.353330 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.353305 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca" pod "3ea0b743-2b8f-416c-a115-e5598bba97f7" (UID: "3ea0b743-2b8f-416c-a115-e5598bba97f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:31.355088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.355065 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4" pod "3ea0b743-2b8f-416c-a115-e5598bba97f7" (UID: "3ea0b743-2b8f-416c-a115-e5598bba97f7"). InnerVolumeSpecName "kube-api-access-q6nq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:31.355183 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.355100 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca" pod "3ea0b743-2b8f-416c-a115-e5598bba97f7" (UID: "3ea0b743-2b8f-416c-a115-e5598bba97f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:31.454014 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.453990 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:31.454014 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.454012 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6nq4\" (UniqueName: \"kubernetes.io/projected/3ea0b743-2b8f-416c-a115-e5598bba97f7-kube-api-access-q6nq4\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:31.454148 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:31.454022 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0b743-2b8f-416c-a115-e5598bba97f7-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:32.137902 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:32.137872 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" Oct 30 22:11:32.138078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:32.137911 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m" event={"ID":"3ea0b743-2b8f-416c-a115-e5598bba97f7","Type":"ContainerDied","Data":"99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d"} Oct 30 22:11:32.138078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:32.137962 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a005975e3a549b8df8275c40377f374d89384ea99a9ad699083f31a7dac01d" Oct 30 22:11:45.803461 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:11:45.803390 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:11:49.434896 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:11:49.434861 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.826864 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:12.826810 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-smwqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:12:12.827585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:12.827532 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-smwqx" podUID="aef085da-a319-4c09-a236-af0467dc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:15.802320 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:12:15.802282 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:12:18.841042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.841011 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj"] Oct 30 22:12:18.841485 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.841472 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ea0b743-2b8f-416c-a115-e5598bba97f7" containerName="revoke-keys" Oct 30 22:12:18.841537 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.841487 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea0b743-2b8f-416c-a115-e5598bba97f7" containerName="revoke-keys" Oct 30 22:12:18.841574 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.841555 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea0b743-2b8f-416c-a115-e5598bba97f7" containerName="revoke-keys" Oct 30 22:12:18.844841 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.844826 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:18.903409 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.903370 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj"] Oct 30 22:12:18.904253 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.904222 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:18.908950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.908890 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95f7\" (UniqueName: \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:18.909292 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.909263 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:18.947672 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.947648 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:18.947807 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:18.947716 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:12:19.010144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.010117 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.010267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.010184 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.010267 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.010214 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95f7\" (UniqueName: \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.010931 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.010910 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.012494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.012471 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.072386 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.072362 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95f7\" (UniqueName: \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\") pod \"maas-api-revoke-sub-e2fc9c5b-zxrhj\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.157519 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.157499 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:19.526662 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:12:19.526629 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e047f6_e0e8_4902_bc38_e57fb9a11f03.slice/crio-42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b WatchSource:0}: Error finding container 42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b: Status 404 returned error can't find the container with id 42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b Oct 30 22:12:19.529283 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:19.529254 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj"] Oct 30 22:12:20.495611 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:20.495580 4918 generic.go:356] "Generic (PLEG): container finished" podID="a8e047f6-e0e8-4902-bc38-e57fb9a11f03" containerID="8c286a4bdd207f515b1a47ac96acd709fd9aef4ab0dd09a7a32bd2628520e758" exitCode=0 Oct 30 22:12:20.495981 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:20.495626 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" event={"ID":"a8e047f6-e0e8-4902-bc38-e57fb9a11f03","Type":"ContainerDied","Data":"8c286a4bdd207f515b1a47ac96acd709fd9aef4ab0dd09a7a32bd2628520e758"} Oct 30 22:12:20.495981 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:20.495675 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" event={"ID":"a8e047f6-e0e8-4902-bc38-e57fb9a11f03","Type":"ContainerStarted","Data":"42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b"} Oct 30 22:12:21.640277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.640249 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:21.733316 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.733289 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\") pod \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " Oct 30 22:12:21.733438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.733346 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\") pod \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " Oct 30 22:12:21.733438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.733376 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\" (UniqueName: \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\") pod \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\" (UID: \"a8e047f6-e0e8-4902-bc38-e57fb9a11f03\") " Oct 30 22:12:21.733691 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.733659 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca" pod "a8e047f6-e0e8-4902-bc38-e57fb9a11f03" (UID: "a8e047f6-e0e8-4902-bc38-e57fb9a11f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:21.735448 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.735419 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca" pod "a8e047f6-e0e8-4902-bc38-e57fb9a11f03" (UID: "a8e047f6-e0e8-4902-bc38-e57fb9a11f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:21.735533 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.735444 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7" pod "a8e047f6-e0e8-4902-bc38-e57fb9a11f03" (UID: "a8e047f6-e0e8-4902-bc38-e57fb9a11f03"). InnerVolumeSpecName "kube-api-access-l95f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:21.834513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.834451 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:21.834513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.834476 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95f7\" (UniqueName: \"kubernetes.io/projected/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-kube-api-access-l95f7\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:21.834513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:21.834486 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e047f6-e0e8-4902-bc38-e57fb9a11f03-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:22.508169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:22.508141 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" Oct 30 22:12:22.508360 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:22.508178 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj" event={"ID":"a8e047f6-e0e8-4902-bc38-e57fb9a11f03","Type":"ContainerDied","Data":"42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b"} Oct 30 22:12:22.508360 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:22.508227 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b5046b841dc3bda35bcea597ec3008364f11bdf2de01a3cafb9ca74ce2e78b" Oct 30 22:12:29.621155 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.621117 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd"] Oct 30 22:12:29.621736 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.621585 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a8e047f6-e0e8-4902-bc38-e57fb9a11f03" containerName="revoke-keys" Oct 30 22:12:29.621736 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.621598 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e047f6-e0e8-4902-bc38-e57fb9a11f03" containerName="revoke-keys" Oct 30 22:12:29.621736 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.621690 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e047f6-e0e8-4902-bc38-e57fb9a11f03" containerName="revoke-keys" Oct 30 22:12:29.625173 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.625150 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.683090 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.683012 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd"] Oct 30 22:12:29.704143 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.704109 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkbg\" (UniqueName: \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.704268 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.704241 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.704317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.704298 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.728455 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.728428 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:29.728614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.728596 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:12:29.805170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.805135 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.805312 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.805218 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkbg\" (UniqueName: \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.805381 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.805313 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.805909 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.805883 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.807581 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.807563 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.868534 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.868511 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkbg\" (UniqueName: \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\") pod \"maas-api-revoke-sub-5a29efb3-9hjrd\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:29.939681 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:29.939621 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:30.315887 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:12:30.315855 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0818d6dc_1cab_44b4_8c57_5f6bc82429ff.slice/crio-90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846 WatchSource:0}: Error finding container 90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846: Status 404 returned error can't find the container with id 90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846 Oct 30 22:12:30.321125 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:30.321095 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd"] Oct 30 22:12:30.558374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:30.558342 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" event={"ID":"0818d6dc-1cab-44b4-8c57-5f6bc82429ff","Type":"ContainerStarted","Data":"90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846"} Oct 30 22:12:31.565776 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:31.565744 4918 generic.go:356] "Generic (PLEG): container finished" podID="0818d6dc-1cab-44b4-8c57-5f6bc82429ff" containerID="daf2cf7075583fdd42167d7a619398b02576c57b15598771efdd3fd987d143b4" exitCode=0 Oct 30 22:12:31.566176 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:31.565832 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" event={"ID":"0818d6dc-1cab-44b4-8c57-5f6bc82429ff","Type":"ContainerDied","Data":"daf2cf7075583fdd42167d7a619398b02576c57b15598771efdd3fd987d143b4"} Oct 30 22:12:32.754812 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.754782 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:32.841930 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.841857 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\") pod \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " Oct 30 22:12:32.842077 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.841981 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\" (UniqueName: \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\") pod \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " Oct 30 22:12:32.842077 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.842043 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\") pod \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\" (UID: \"0818d6dc-1cab-44b4-8c57-5f6bc82429ff\") " Oct 30 22:12:32.842522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.842484 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca" pod "0818d6dc-1cab-44b4-8c57-5f6bc82429ff" (UID: "0818d6dc-1cab-44b4-8c57-5f6bc82429ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:32.844009 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.843983 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca" pod "0818d6dc-1cab-44b4-8c57-5f6bc82429ff" (UID: "0818d6dc-1cab-44b4-8c57-5f6bc82429ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:32.844580 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.844556 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg" pod "0818d6dc-1cab-44b4-8c57-5f6bc82429ff" (UID: "0818d6dc-1cab-44b4-8c57-5f6bc82429ff"). InnerVolumeSpecName "kube-api-access-ddkbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:32.943504 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.943468 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddkbg\" (UniqueName: \"kubernetes.io/projected/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-kube-api-access-ddkbg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:32.943662 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.943508 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:32.943662 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:32.943526 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818d6dc-1cab-44b4-8c57-5f6bc82429ff-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:33.607729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:33.607199 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" event={"ID":"0818d6dc-1cab-44b4-8c57-5f6bc82429ff","Type":"ContainerDied","Data":"90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846"} Oct 30 22:12:33.607729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:33.607268 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90bc5cb9f6724e8a63d175772d1f49a7bc60b7789fe590ad9d074a2b9e646846" Oct 30 22:12:33.607729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:33.607341 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd" Oct 30 22:12:45.802272 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:12:45.802234 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:12:46.994890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:46.994848 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m"] Oct 30 22:12:47.043507 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.043468 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gjm2m"] Oct 30 22:12:47.117546 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.117515 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx"] Oct 30 22:12:47.118046 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.118031 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0818d6dc-1cab-44b4-8c57-5f6bc82429ff" containerName="revoke-keys" Oct 30 22:12:47.118102 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.118048 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0818d6dc-1cab-44b4-8c57-5f6bc82429ff" containerName="revoke-keys" Oct 30 22:12:47.118144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.118131 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0818d6dc-1cab-44b4-8c57-5f6bc82429ff" containerName="revoke-keys" Oct 30 22:12:47.121433 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.121418 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.174320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.174289 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.174461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.174425 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.174526 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.174510 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxv4c\" (UniqueName: \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.175851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.175820 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:12:47.175851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.175821 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:47.177620 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.177595 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx"] Oct 30 22:12:47.275768 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.275670 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxv4c\" (UniqueName: \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.275890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.275772 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.275965 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.275887 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.276480 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.276456 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.278130 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.278111 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.337753 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.337725 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxv4c\" (UniqueName: \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\") pod \"maas-api-revoke-sub-3d4156f1-69xvx\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.427287 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.427261 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea0b743-2b8f-416c-a115-e5598bba97f7" path="/var/lib/kubelet/pods/3ea0b743-2b8f-416c-a115-e5598bba97f7/volumes" Oct 30 22:12:47.434473 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.434456 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:47.813094 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:47.813059 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx"] Oct 30 22:12:48.724465 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:48.724435 4918 generic.go:356] "Generic (PLEG): container finished" podID="07d9e833-da67-40ac-b102-d7a03805d94e" containerID="11244aa717243655b86ea47e2dc837ffc031349cc3d040c18920cfd1d2b69d1a" exitCode=0 Oct 30 22:12:48.724822 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:48.724521 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" event={"ID":"07d9e833-da67-40ac-b102-d7a03805d94e","Type":"ContainerDied","Data":"11244aa717243655b86ea47e2dc837ffc031349cc3d040c18920cfd1d2b69d1a"} Oct 30 22:12:48.724822 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:48.724573 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" event={"ID":"07d9e833-da67-40ac-b102-d7a03805d94e","Type":"ContainerStarted","Data":"e825aa640714f4a0f9c058193d3d734547129858265a16b7ad4d98f4c020fef7"} Oct 30 22:12:49.745250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:49.745221 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.870354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:49.870332 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:50.001858 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.001762 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\") pod \"07d9e833-da67-40ac-b102-d7a03805d94e\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " Oct 30 22:12:50.002016 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.001875 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\") pod \"07d9e833-da67-40ac-b102-d7a03805d94e\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " Oct 30 22:12:50.002016 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.001951 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\" (UniqueName: \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\") pod \"07d9e833-da67-40ac-b102-d7a03805d94e\" (UID: \"07d9e833-da67-40ac-b102-d7a03805d94e\") " Oct 30 22:12:50.002295 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.002255 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca" pod "07d9e833-da67-40ac-b102-d7a03805d94e" (UID: "07d9e833-da67-40ac-b102-d7a03805d94e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:50.004169 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.004144 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca" pod "07d9e833-da67-40ac-b102-d7a03805d94e" (UID: "07d9e833-da67-40ac-b102-d7a03805d94e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:50.005064 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.005040 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c" pod "07d9e833-da67-40ac-b102-d7a03805d94e" (UID: "07d9e833-da67-40ac-b102-d7a03805d94e"). InnerVolumeSpecName "kube-api-access-bxv4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:50.103359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.103333 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:50.103359 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.103357 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxv4c\" (UniqueName: \"kubernetes.io/projected/07d9e833-da67-40ac-b102-d7a03805d94e-kube-api-access-bxv4c\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:50.103522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.103369 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d9e833-da67-40ac-b102-d7a03805d94e-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:50.738676 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.738645 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" Oct 30 22:12:50.738911 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.738645 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx" event={"ID":"07d9e833-da67-40ac-b102-d7a03805d94e","Type":"ContainerDied","Data":"e825aa640714f4a0f9c058193d3d734547129858265a16b7ad4d98f4c020fef7"} Oct 30 22:12:50.738911 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:12:50.738773 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e825aa640714f4a0f9c058193d3d734547129858265a16b7ad4d98f4c020fef7" Oct 30 22:13:15.802747 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:13:15.802683 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:13:19.151860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:19.151825 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:19.774047 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:19.774015 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:20.096179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:20.096089 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:24.233999 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:24.233962 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:24.499451 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:24.499378 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.579275 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:27.579232 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.769836 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:27.769794 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:30.671632 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.671600 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:13:30.672122 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.672063 4918 cpu_manager.go:414] "containerMap: removing container" podUID="07d9e833-da67-40ac-b102-d7a03805d94e" containerName="revoke-keys" Oct 30 22:13:30.672122 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.672076 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d9e833-da67-40ac-b102-d7a03805d94e" containerName="revoke-keys" Oct 30 22:13:30.672202 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.672168 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d9e833-da67-40ac-b102-d7a03805d94e" containerName="revoke-keys" Oct 30 22:13:30.675611 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.675591 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.733912 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.733879 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:13:30.767063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.767033 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-788e42-serving-cert\"" Oct 30 22:13:30.780197 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.780163 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hkklw\"" Oct 30 22:13:30.780197 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.780174 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:13:30.792570 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.792548 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkckl\" (UniqueName: \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.792671 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.792594 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.792769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.792675 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.792769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.792752 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.864298 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.864263 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:30.894257 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.894226 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.894414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.894287 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.894414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.894314 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.894414 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.894371 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkckl\" (UniqueName: \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.895025 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.895003 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.896749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.896727 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.896853 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.896837 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.957077 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.957024 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkckl\" (UniqueName: \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\") pod \"maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:30.988899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:30.988872 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:31.053138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:31.053104 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.374506 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:13:31.374467 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7758e50_5989_48c3_bfb4_8749c5916c8e.slice/crio-5ecb938e48ec34bb6949bcdc767e7e1292498af245cc25399e6dd271f7b2e15a WatchSource:0}: Error finding container 5ecb938e48ec34bb6949bcdc767e7e1292498af245cc25399e6dd271f7b2e15a: Status 404 returned error can't find the container with id 5ecb938e48ec34bb6949bcdc767e7e1292498af245cc25399e6dd271f7b2e15a Oct 30 22:13:31.376027 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:31.376005 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:13:31.377523 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:31.377501 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:13:32.029362 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:32.029329 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" event={"ID":"a7758e50-5989-48c3-bfb4-8749c5916c8e","Type":"ContainerStarted","Data":"5ecb938e48ec34bb6949bcdc767e7e1292498af245cc25399e6dd271f7b2e15a"} Oct 30 22:13:33.037347 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:33.037305 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" event={"ID":"a7758e50-5989-48c3-bfb4-8749c5916c8e","Type":"ContainerStarted","Data":"d1cb46aacaf0222b5755b6ce6622bb888d32186e4f09d8e69be4a0af1dabced4"} Oct 30 22:13:33.037779 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:33.037397 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:33.157976 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:33.157929 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" podStartSLOduration=2.3232418519999998 podStartE2EDuration="3.157915358s" totalImagesPullingTime="834.673506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:33.156909289 +0000 UTC m=+3593.850391525" watchObservedRunningTime="2026-10-30 22:13:33.157915358 +0000 UTC m=+3593.851397594" Oct 30 22:13:35.892510 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:35.892472 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:38.053945 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:38.053909 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d"] Oct 30 22:13:38.104504 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:38.104474 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zdz4d"] Oct 30 22:13:39.048578 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:39.048551 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:13:39.428467 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:39.428433 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94" path="/var/lib/kubelet/pods/b95e6b0c-f27e-44bc-97ba-ee17a0c8fd94/volumes" Oct 30 22:13:40.015432 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.015403 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:13:40.025989 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.025968 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:13:40.029781 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.029757 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:13:40.035483 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.035461 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:13:40.040219 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.040201 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:13:40.045453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.045433 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:13:40.918484 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:40.918452 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:45.801679 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:13:45.801641 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:13:46.035276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:46.035243 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.065840 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:49.065798 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l"] Oct 30 22:13:49.116593 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:49.116552 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x5k6l"] Oct 30 22:13:49.129019 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:49.128993 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.427833 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:49.427802 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857e6222-7e5b-4372-be1b-b162487fc04a" path="/var/lib/kubelet/pods/857e6222-7e5b-4372-be1b-b162487fc04a/volumes" Oct 30 22:13:49.844377 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:49.844302 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.069742 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:50.069687 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.294517 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:50.294489 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.508553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:50.508516 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.425851 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:51.425821 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.273389 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.273354 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:13:53.278567 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.278543 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.334311 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.334278 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:13:53.380661 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.380634 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-788e42\"/\"simulator-single-main-788e42-kserve-self-signed-certs\"" Oct 30 22:13:53.380839 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.380716 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-788e42\"/\"openshift-service-ca.crt\"" Oct 30 22:13:53.380839 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.380822 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-788e42\"/\"default-dockercfg-5crvt\"" Oct 30 22:13:53.385569 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.385546 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-788e42\"/\"kube-root-ca.crt\"" Oct 30 22:13:53.395895 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.395877 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.395975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.395926 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.396022 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.396005 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgq98\" (UniqueName: \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.396063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.396042 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.396098 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.396063 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.396133 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.396109 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.414290 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.414264 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.496987 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.496956 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497188 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.496998 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497188 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497021 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497201 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497280 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497336 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgq98\" (UniqueName: \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497410 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497523 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497494 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.497772 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.497747 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.499234 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.499216 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.499634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.499616 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.558910 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.558847 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgq98\" (UniqueName: \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\") pod \"simulator-single-main-788e42-kserve-644f978cdd-j9sxd\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.591337 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.591314 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:53.961627 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:13:53.961593 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94c561b1_22a2_4527_8a4d_61a2777b450a.slice/crio-43128dd11e70e2af5c63b7e844a6ef18b9c054277d90fe0545865d5f4073aa93 WatchSource:0}: Error finding container 43128dd11e70e2af5c63b7e844a6ef18b9c054277d90fe0545865d5f4073aa93: Status 404 returned error can't find the container with id 43128dd11e70e2af5c63b7e844a6ef18b9c054277d90fe0545865d5f4073aa93 Oct 30 22:13:53.964941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:53.964911 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:13:54.190164 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:54.190128 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" event={"ID":"94c561b1-22a2-4527-8a4d-61a2777b450a","Type":"ContainerStarted","Data":"73074b566b7af08eaa39cb91aeb4a3fad7515b67e1c94cc3dd68c2f6e23f0bfe"} Oct 30 22:13:54.190164 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:54.190174 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" event={"ID":"94c561b1-22a2-4527-8a4d-61a2777b450a","Type":"ContainerStarted","Data":"43128dd11e70e2af5c63b7e844a6ef18b9c054277d90fe0545865d5f4073aa93"} Oct 30 22:13:54.333606 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:54.333517 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" podStartSLOduration=1.333506591 podStartE2EDuration="1.333506591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:54.333360248 +0000 UTC m=+3615.026842486" watchObservedRunningTime="2026-10-30 22:13:54.333506591 +0000 UTC m=+3615.026988826" Oct 30 22:13:54.591758 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:54.591665 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:13:56.530584 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:13:56.530544 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:01.479652 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:01.479614 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:03.592390 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:03.592360 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:14:03.605059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:03.605029 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:14:04.271348 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:04.271316 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:14:06.464587 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:06.464543 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.390824 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:11.390767 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.802430 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:14:15.802389 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:14:16.275201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:16.273786 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.282068 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:21.281943 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:24.065515 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:24.065483 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h"] Oct 30 22:14:24.115636 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:24.115609 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z6h"] Oct 30 22:14:25.427958 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:25.427923 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741ff9d6-feb0-4a26-ba42-cfc36413a61e" path="/var/lib/kubelet/pods/741ff9d6-feb0-4a26-ba42-cfc36413a61e/volumes" Oct 30 22:14:26.393418 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:26.393380 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.389270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:31.389227 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.479471 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:36.479396 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.486861 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:41.486827 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:44.338986 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:44.338956 4918 scope.go:122] "RemoveContainer" containerID="fd085651bdbb5ab8c4ac7068e254c897d51a9b813c2032f10cedc41c586487c9" Oct 30 22:14:45.802868 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:14:45.802831 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:14:46.486759 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:46.486693 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.689846 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:49.689795 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:50.458255 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:50.458221 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:51.739821 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:51.739791 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.052177 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:52.052107 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s"] Oct 30 22:14:52.102352 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:52.102324 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fzz8s"] Oct 30 22:14:53.427810 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:53.427774 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb98266-69db-4f7f-b0e7-1d44e0bb6179" path="/var/lib/kubelet/pods/9bb98266-69db-4f7f-b0e7-1d44e0bb6179/volumes" Oct 30 22:14:54.541976 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:54.541943 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:14:54.666489 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:54.666456 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.832329 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:56.832298 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:57.105948 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:14:57.105855 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:01.440719 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.440669 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:15:01.441302 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.440919 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" podUID="a7758e50-5989-48c3-bfb4-8749c5916c8e" containerName="maas-api" containerID="cri-o://d1cb46aacaf0222b5755b6ce6622bb888d32186e4f09d8e69be4a0af1dabced4" gracePeriod=30 Oct 30 22:15:01.565675 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.565648 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:01.639758 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.639723 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7758e50-5989-48c3-bfb4-8749c5916c8e" containerID="d1cb46aacaf0222b5755b6ce6622bb888d32186e4f09d8e69be4a0af1dabced4" exitCode=0 Oct 30 22:15:01.639895 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.639818 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" event={"ID":"a7758e50-5989-48c3-bfb4-8749c5916c8e","Type":"ContainerDied","Data":"d1cb46aacaf0222b5755b6ce6622bb888d32186e4f09d8e69be4a0af1dabced4"} Oct 30 22:15:01.724876 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.724849 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:15:01.824696 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.824662 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\") pod \"a7758e50-5989-48c3-bfb4-8749c5916c8e\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " Oct 30 22:15:01.824919 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.824820 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\" (UniqueName: \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\") pod \"a7758e50-5989-48c3-bfb4-8749c5916c8e\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " Oct 30 22:15:01.824919 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.824849 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\") pod \"a7758e50-5989-48c3-bfb4-8749c5916c8e\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " Oct 30 22:15:01.824919 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.824892 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\") pod \"a7758e50-5989-48c3-bfb4-8749c5916c8e\" (UID: \"a7758e50-5989-48c3-bfb4-8749c5916c8e\") " Oct 30 22:15:01.825748 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.825698 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca" pod "a7758e50-5989-48c3-bfb4-8749c5916c8e" (UID: "a7758e50-5989-48c3-bfb4-8749c5916c8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:01.829746 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.827441 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs" pod "a7758e50-5989-48c3-bfb4-8749c5916c8e" (UID: "a7758e50-5989-48c3-bfb4-8749c5916c8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:01.829746 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.827572 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls" pod "a7758e50-5989-48c3-bfb4-8749c5916c8e" (UID: "a7758e50-5989-48c3-bfb4-8749c5916c8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:01.829746 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.827684 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl" pod "a7758e50-5989-48c3-bfb4-8749c5916c8e" (UID: "a7758e50-5989-48c3-bfb4-8749c5916c8e"). InnerVolumeSpecName "kube-api-access-xkckl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:01.926146 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.926117 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:01.926146 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.926143 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkckl\" (UniqueName: \"kubernetes.io/projected/a7758e50-5989-48c3-bfb4-8749c5916c8e-kube-api-access-xkckl\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:01.926146 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.926155 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7758e50-5989-48c3-bfb4-8749c5916c8e-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:01.926380 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:01.926165 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7758e50-5989-48c3-bfb4-8749c5916c8e-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:02.153654 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.153623 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:02.394626 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.394598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:02.647310 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.647282 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" Oct 30 22:15:02.647752 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.647307 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg" event={"ID":"a7758e50-5989-48c3-bfb4-8749c5916c8e","Type":"ContainerDied","Data":"5ecb938e48ec34bb6949bcdc767e7e1292498af245cc25399e6dd271f7b2e15a"} Oct 30 22:15:02.647752 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.647367 4918 scope.go:122] "RemoveContainer" containerID="d1cb46aacaf0222b5755b6ce6622bb888d32186e4f09d8e69be4a0af1dabced4" Oct 30 22:15:02.825282 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.825252 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:15:02.875537 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:02.875509 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-788e42-8665f99d5b-8h6rg"] Oct 30 22:15:03.428073 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:03.428035 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7758e50-5989-48c3-bfb4-8749c5916c8e" path="/var/lib/kubelet/pods/a7758e50-5989-48c3-bfb4-8749c5916c8e/volumes" Oct 30 22:15:06.224766 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:06.224732 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.518076 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:06.517999 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.370586 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:07.370550 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.562271 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.562232 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" podUID="94c561b1-22a2-4527-8a4d-61a2777b450a" containerName="main" containerID="cri-o://73074b566b7af08eaa39cb91aeb4a3fad7515b67e1c94cc3dd68c2f6e23f0bfe" gracePeriod=45 Oct 30 22:15:09.694100 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.694077 4918 generic.go:356] "Generic (PLEG): container finished" podID="94c561b1-22a2-4527-8a4d-61a2777b450a" containerID="73074b566b7af08eaa39cb91aeb4a3fad7515b67e1c94cc3dd68c2f6e23f0bfe" exitCode=0 Oct 30 22:15:09.694213 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.694150 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" event={"ID":"94c561b1-22a2-4527-8a4d-61a2777b450a","Type":"ContainerDied","Data":"73074b566b7af08eaa39cb91aeb4a3fad7515b67e1c94cc3dd68c2f6e23f0bfe"} Oct 30 22:15:09.829144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.829068 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:15:09.896770 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896749 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.896888 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896793 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\" (UniqueName: \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.896888 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896817 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.897023 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896895 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\" (UniqueName: \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.897023 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896912 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.897023 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.896942 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\") pod \"94c561b1-22a2-4527-8a4d-61a2777b450a\" (UID: \"94c561b1-22a2-4527-8a4d-61a2777b450a\") " Oct 30 22:15:09.897187 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.897010 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:09.897187 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.897113 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:09.897284 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.897240 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:09.897284 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.897269 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:09.897354 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.897290 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:09.898993 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.898970 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:09.899097 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.899079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:09.899229 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.899208 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98" pod "94c561b1-22a2-4527-8a4d-61a2777b450a" (UID: "94c561b1-22a2-4527-8a4d-61a2777b450a"). InnerVolumeSpecName "kube-api-access-xgq98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:09.998562 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.998518 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgq98\" (UniqueName: \"kubernetes.io/projected/94c561b1-22a2-4527-8a4d-61a2777b450a-kube-api-access-xgq98\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:09.998562 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.998564 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:09.998778 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.998584 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c561b1-22a2-4527-8a4d-61a2777b450a-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:09.998778 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:09.998601 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c561b1-22a2-4527-8a4d-61a2777b450a-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.703868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:10.703824 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" event={"ID":"94c561b1-22a2-4527-8a4d-61a2777b450a","Type":"ContainerDied","Data":"43128dd11e70e2af5c63b7e844a6ef18b9c054277d90fe0545865d5f4073aa93"} Oct 30 22:15:10.704378 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:10.703880 4918 scope.go:122] "RemoveContainer" containerID="73074b566b7af08eaa39cb91aeb4a3fad7515b67e1c94cc3dd68c2f6e23f0bfe" Oct 30 22:15:10.704378 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:10.703846 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd" Oct 30 22:15:10.872036 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:10.872002 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:15:10.922849 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:10.922817 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-788e42/simulator-single-main-788e42-kserve-644f978cdd-j9sxd"] Oct 30 22:15:11.427798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:11.427761 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c561b1-22a2-4527-8a4d-61a2777b450a" path="/var/lib/kubelet/pods/94c561b1-22a2-4527-8a4d-61a2777b450a/volumes" Oct 30 22:15:13.381462 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:13.381428 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:13.865867 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:13.865826 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:14.054136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:14.054100 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:14.277642 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:14.277543 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.804354 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:15:15.804317 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:15:15.889491 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:15.889461 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx"] Oct 30 22:15:15.940178 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:15.940148 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-69xvx"] Oct 30 22:15:16.014042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014015 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv"] Oct 30 22:15:16.014450 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014439 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7758e50-5989-48c3-bfb4-8749c5916c8e" containerName="maas-api" Oct 30 22:15:16.014502 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014452 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7758e50-5989-48c3-bfb4-8749c5916c8e" containerName="maas-api" Oct 30 22:15:16.014539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014506 4918 cpu_manager.go:414] "containerMap: removing container" podUID="94c561b1-22a2-4527-8a4d-61a2777b450a" containerName="main" Oct 30 22:15:16.014539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014513 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c561b1-22a2-4527-8a4d-61a2777b450a" containerName="main" Oct 30 22:15:16.014602 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014580 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c561b1-22a2-4527-8a4d-61a2777b450a" containerName="main" Oct 30 22:15:16.014602 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.014589 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7758e50-5989-48c3-bfb4-8749c5916c8e" containerName="maas-api" Oct 30 22:15:16.017951 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.017932 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.054460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.054404 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.054460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.054440 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.054619 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.054581 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj6q\" (UniqueName: \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.071463 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.071426 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:16.071587 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.071566 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:15:16.093138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.093114 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv"] Oct 30 22:15:16.155083 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.155057 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.155246 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.155163 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj6q\" (UniqueName: \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.155246 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.155226 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.155838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.155816 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.157676 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.157650 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.218355 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.218323 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj6q\" (UniqueName: \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\") pod \"maas-api-revoke-sub-3d4156f1-xsnkv\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.331182 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.331103 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:16.709822 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:15:16.709782 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168ecedc_1940_4610_b18a_a32a80762d36.slice/crio-e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a WatchSource:0}: Error finding container e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a: Status 404 returned error can't find the container with id e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a Oct 30 22:15:16.713470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.713394 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv"] Oct 30 22:15:16.743801 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:16.743775 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" event={"ID":"168ecedc-1940-4610-b18a-a32a80762d36","Type":"ContainerStarted","Data":"e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a"} Oct 30 22:15:17.427734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:17.427685 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d9e833-da67-40ac-b102-d7a03805d94e" path="/var/lib/kubelet/pods/07d9e833-da67-40ac-b102-d7a03805d94e/volumes" Oct 30 22:15:17.751795 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:17.751730 4918 generic.go:356] "Generic (PLEG): container finished" podID="168ecedc-1940-4610-b18a-a32a80762d36" containerID="cf92504199dd218a6de5a8aed3eb09dbaabed911b5913c0ee07978575e02e6aa" exitCode=0 Oct 30 22:15:17.751985 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:17.751801 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" event={"ID":"168ecedc-1940-4610-b18a-a32a80762d36","Type":"ContainerDied","Data":"cf92504199dd218a6de5a8aed3eb09dbaabed911b5913c0ee07978575e02e6aa"} Oct 30 22:15:18.898433 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.898408 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:18.984209 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.984184 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\" (UniqueName: \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\") pod \"168ecedc-1940-4610-b18a-a32a80762d36\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " Oct 30 22:15:18.984363 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.984250 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\") pod \"168ecedc-1940-4610-b18a-a32a80762d36\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " Oct 30 22:15:18.984363 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.984352 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\") pod \"168ecedc-1940-4610-b18a-a32a80762d36\" (UID: \"168ecedc-1940-4610-b18a-a32a80762d36\") " Oct 30 22:15:18.984747 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.984718 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca" pod "168ecedc-1940-4610-b18a-a32a80762d36" (UID: "168ecedc-1940-4610-b18a-a32a80762d36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.986080 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.986056 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q" pod "168ecedc-1940-4610-b18a-a32a80762d36" (UID: "168ecedc-1940-4610-b18a-a32a80762d36"). InnerVolumeSpecName "kube-api-access-gwj6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.986171 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:18.986111 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca" pod "168ecedc-1940-4610-b18a-a32a80762d36" (UID: "168ecedc-1940-4610-b18a-a32a80762d36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:19.085764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.085658 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.085764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.085695 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwj6q\" (UniqueName: \"kubernetes.io/projected/168ecedc-1940-4610-b18a-a32a80762d36-kube-api-access-gwj6q\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.085764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.085731 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168ecedc-1940-4610-b18a-a32a80762d36-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.766018 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.765937 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" Oct 30 22:15:19.766018 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.765972 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv" event={"ID":"168ecedc-1940-4610-b18a-a32a80762d36","Type":"ContainerDied","Data":"e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a"} Oct 30 22:15:19.766018 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:19.766016 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45a55b31f1171720a8b738bde923ab7cd53318f283a2fb312820c89fd63937a" Oct 30 22:15:23.058538 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:23.058500 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd"] Oct 30 22:15:23.104379 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:23.104349 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7tshd"] Oct 30 22:15:23.427621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:23.427582 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814e49e3-6e6f-4985-b595-e2f9b37ee0ac" path="/var/lib/kubelet/pods/814e49e3-6e6f-4985-b595-e2f9b37ee0ac/volumes" Oct 30 22:15:31.749277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:31.749241 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:33.117393 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:33.117362 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:38.415833 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:38.415792 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:44.363503 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:44.363468 4918 scope.go:122] "RemoveContainer" containerID="080251e4841e94724bc41704722af996eed653a122e831e1fea6a8d428da66db" Oct 30 22:15:44.378488 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:44.378457 4918 scope.go:122] "RemoveContainer" containerID="9b77bcc2c3d816681fa0b3c1053a646f9fb1750dc343cba628fc6a99e158bc35" Oct 30 22:15:45.805835 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:15:45.805793 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:15:46.060262 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:46.060187 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq"] Oct 30 22:15:46.110642 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:46.110608 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ql4cq"] Oct 30 22:15:47.428035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:47.427976 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282d0cec-79e3-4d2b-8d24-70e0930d7e8a" path="/var/lib/kubelet/pods/282d0cec-79e3-4d2b-8d24-70e0930d7e8a/volumes" Oct 30 22:15:50.769857 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:50.769817 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 22:15:50.770398 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:50.770010 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" podUID="db2f906f-cf82-44c1-a32c-ace501143a05" containerName="manager" containerID="cri-o://6e8273a2240c089b6f837421a0c5a1eac820e8e27f6bbc7d158d4442e21b9ec0" gracePeriod=10 Oct 30 22:15:50.796113 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:50.796088 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.985116 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:50.985073 4918 generic.go:356] "Generic (PLEG): container finished" podID="db2f906f-cf82-44c1-a32c-ace501143a05" containerID="6e8273a2240c089b6f837421a0c5a1eac820e8e27f6bbc7d158d4442e21b9ec0" exitCode=0 Oct 30 22:15:50.985292 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:50.985165 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" event={"ID":"db2f906f-cf82-44c1-a32c-ace501143a05","Type":"ContainerDied","Data":"6e8273a2240c089b6f837421a0c5a1eac820e8e27f6bbc7d158d4442e21b9ec0"} Oct 30 22:15:51.047353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.047330 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 22:15:51.220218 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.220182 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\" (UniqueName: \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\") pod \"db2f906f-cf82-44c1-a32c-ace501143a05\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " Oct 30 22:15:51.220403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.220276 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\") pod \"db2f906f-cf82-44c1-a32c-ace501143a05\" (UID: \"db2f906f-cf82-44c1-a32c-ace501143a05\") " Oct 30 22:15:51.221059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.221023 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume" pod "db2f906f-cf82-44c1-a32c-ace501143a05" (UID: "db2f906f-cf82-44c1-a32c-ace501143a05"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.235985 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.235954 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc" pod "db2f906f-cf82-44c1-a32c-ace501143a05" (UID: "db2f906f-cf82-44c1-a32c-ace501143a05"). InnerVolumeSpecName "kube-api-access-fqjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:51.321247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.321173 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjgc\" (UniqueName: \"kubernetes.io/projected/db2f906f-cf82-44c1-a32c-ace501143a05-kube-api-access-fqjgc\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.321247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.321202 4918 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db2f906f-cf82-44c1-a32c-ace501143a05-extensions-socket-volume\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.992872 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.992835 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" event={"ID":"db2f906f-cf82-44c1-a32c-ace501143a05","Type":"ContainerDied","Data":"1ec3ebd0ddb473d5fb356962253b9d94c0605f38f2c712d2a23e6be5e269b36d"} Oct 30 22:15:51.993275 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.992896 4918 scope.go:122] "RemoveContainer" containerID="6e8273a2240c089b6f837421a0c5a1eac820e8e27f6bbc7d158d4442e21b9ec0" Oct 30 22:15:51.993275 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:51.992851 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k" Oct 30 22:15:52.212657 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:52.212629 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 22:15:52.263453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:52.263402 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l6x6k"] Oct 30 22:15:53.427440 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:15:53.427398 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2f906f-cf82-44c1-a32c-ace501143a05" path="/var/lib/kubelet/pods/db2f906f-cf82-44c1-a32c-ace501143a05/volumes" Oct 30 22:16:15.801735 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:16:15.801676 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:16:19.784924 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:19.784868 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.418304 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:44.418265 4918 scope.go:122] "RemoveContainer" containerID="12c3f16724b09d190b7a903778d96bd22245b889e36c0a359c52c561ec5cf59a" Oct 30 22:16:44.427997 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:44.427974 4918 scope.go:122] "RemoveContainer" containerID="07dd74e43a3d099a96111ca3d8f9d43ea1e3096336b5236b34d019104f8bb914" Oct 30 22:16:44.436423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:44.436404 4918 scope.go:122] "RemoveContainer" containerID="e41c0df01141dd2789c75584a0821c4ba26c6ca0283574eb39733168585e299b" Oct 30 22:16:45.801013 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:16:45.800974 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:16:51.176367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:51.176331 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:51.432688 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:51.432607 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:51.643592 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:51.643562 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:54.394853 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.394820 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5"] Oct 30 22:16:54.395260 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395247 4918 cpu_manager.go:414] "containerMap: removing container" podUID="168ecedc-1940-4610-b18a-a32a80762d36" containerName="revoke-keys" Oct 30 22:16:54.395304 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395260 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168ecedc-1940-4610-b18a-a32a80762d36" containerName="revoke-keys" Oct 30 22:16:54.395304 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395300 4918 cpu_manager.go:414] "containerMap: removing container" podUID="db2f906f-cf82-44c1-a32c-ace501143a05" containerName="manager" Oct 30 22:16:54.395374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395305 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f906f-cf82-44c1-a32c-ace501143a05" containerName="manager" Oct 30 22:16:54.395374 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395370 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f906f-cf82-44c1-a32c-ace501143a05" containerName="manager" Oct 30 22:16:54.395446 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.395381 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="168ecedc-1940-4610-b18a-a32a80762d36" containerName="revoke-keys" Oct 30 22:16:54.398631 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.398607 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.474719 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.474678 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bbe384e-e4d3-4917-9985-7fbb079f5e04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.474848 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.474729 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/4bbe384e-e4d3-4917-9985-7fbb079f5e04-kube-api-access-2nr6t\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.514446 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.514417 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vbmj6\"" Oct 30 22:16:54.517504 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.517485 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5"] Oct 30 22:16:54.575351 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.575328 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bbe384e-e4d3-4917-9985-7fbb079f5e04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.575443 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.575359 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/4bbe384e-e4d3-4917-9985-7fbb079f5e04-kube-api-access-2nr6t\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.575677 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.575657 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bbe384e-e4d3-4917-9985-7fbb079f5e04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.638062 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.638039 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nr6t\" (UniqueName: \"kubernetes.io/projected/4bbe384e-e4d3-4917-9985-7fbb079f5e04-kube-api-access-2nr6t\") pod \"kuadrant-operator-controller-manager-678d87766f-65rf5\" (UID: \"4bbe384e-e4d3-4917-9985-7fbb079f5e04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:54.711524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:54.711473 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:55.185652 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:16:55.185619 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbe384e_e4d3_4917_9985_7fbb079f5e04.slice/crio-96aa6ec3c28dfe7855c95548700f402e82c4428ff5c3818c4e58690d324b8702 WatchSource:0}: Error finding container 96aa6ec3c28dfe7855c95548700f402e82c4428ff5c3818c4e58690d324b8702: Status 404 returned error can't find the container with id 96aa6ec3c28dfe7855c95548700f402e82c4428ff5c3818c4e58690d324b8702 Oct 30 22:16:55.237962 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:55.237937 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5"] Oct 30 22:16:55.388960 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:55.388924 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" event={"ID":"4bbe384e-e4d3-4917-9985-7fbb079f5e04","Type":"ContainerStarted","Data":"9be8745ed100fdb1f3b6c36c078eaf6c285df4a7727bb1beea89771f5699e515"} Oct 30 22:16:55.389114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:55.388971 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" event={"ID":"4bbe384e-e4d3-4917-9985-7fbb079f5e04","Type":"ContainerStarted","Data":"96aa6ec3c28dfe7855c95548700f402e82c4428ff5c3818c4e58690d324b8702"} Oct 30 22:16:55.389114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:55.389056 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:16:55.512300 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:16:55.512215 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" podStartSLOduration=1.512202386 podStartE2EDuration="1.512202386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:55.510562667 +0000 UTC m=+3796.204044903" watchObservedRunningTime="2026-10-30 22:16:55.512202386 +0000 UTC m=+3796.205684621" Oct 30 22:17:06.397174 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:06.397139 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-65rf5" Oct 30 22:17:15.803805 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:17:15.803763 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:17:20.160979 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.160892 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t"] Oct 30 22:17:20.164936 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.164911 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.219353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.219299 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t"] Oct 30 22:17:20.240897 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.240870 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.241083 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.240910 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.241083 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.241033 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dccl\" (UniqueName: \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.267138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.267113 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:20.267253 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.267211 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:17:20.342505 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.342475 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dccl\" (UniqueName: \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.342641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.342550 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.342641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.342577 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.343257 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.343236 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.344950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.344930 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.405835 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.405811 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dccl\" (UniqueName: \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\") pod \"maas-api-revoke-sub-d576e68b-5jj8t\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.480738 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.480667 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:20.852282 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:17:20.852246 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361af7e8_ecd3_40f3_8742_d96cd1736feb.slice/crio-cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb WatchSource:0}: Error finding container cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb: Status 404 returned error can't find the container with id cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb Oct 30 22:17:20.854168 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:20.854145 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t"] Oct 30 22:17:21.564941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:21.564862 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" event={"ID":"361af7e8-ecd3-40f3-8742-d96cd1736feb","Type":"ContainerStarted","Data":"8c6e42607388209a7e9270aecd48aabbbc74abdb6e5c2440e38929ab7da40892"} Oct 30 22:17:21.564941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:21.564914 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" event={"ID":"361af7e8-ecd3-40f3-8742-d96cd1736feb","Type":"ContainerStarted","Data":"cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb"} Oct 30 22:17:21.689478 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:21.689435 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" podStartSLOduration=1.689423694 podStartE2EDuration="1.689423694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:21.688826901 +0000 UTC m=+3822.382309138" watchObservedRunningTime="2026-10-30 22:17:21.689423694 +0000 UTC m=+3822.382905929" Oct 30 22:17:22.060492 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:22.060455 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj"] Oct 30 22:17:22.063357 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:22.063336 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zxrhj"] Oct 30 22:17:22.572765 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:22.572731 4918 generic.go:356] "Generic (PLEG): container finished" podID="361af7e8-ecd3-40f3-8742-d96cd1736feb" containerID="8c6e42607388209a7e9270aecd48aabbbc74abdb6e5c2440e38929ab7da40892" exitCode=0 Oct 30 22:17:22.573163 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:22.572808 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" event={"ID":"361af7e8-ecd3-40f3-8742-d96cd1736feb","Type":"ContainerDied","Data":"8c6e42607388209a7e9270aecd48aabbbc74abdb6e5c2440e38929ab7da40892"} Oct 30 22:17:23.428065 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.428032 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e047f6-e0e8-4902-bc38-e57fb9a11f03" path="/var/lib/kubelet/pods/a8e047f6-e0e8-4902-bc38-e57fb9a11f03/volumes" Oct 30 22:17:23.710088 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.710065 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:23.874616 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.874587 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\") pod \"361af7e8-ecd3-40f3-8742-d96cd1736feb\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " Oct 30 22:17:23.874838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.874743 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\") pod \"361af7e8-ecd3-40f3-8742-d96cd1736feb\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " Oct 30 22:17:23.874838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.874777 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\" (UniqueName: \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\") pod \"361af7e8-ecd3-40f3-8742-d96cd1736feb\" (UID: \"361af7e8-ecd3-40f3-8742-d96cd1736feb\") " Oct 30 22:17:23.875038 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.875001 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca" pod "361af7e8-ecd3-40f3-8742-d96cd1736feb" (UID: "361af7e8-ecd3-40f3-8742-d96cd1736feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:23.876716 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.876682 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca" pod "361af7e8-ecd3-40f3-8742-d96cd1736feb" (UID: "361af7e8-ecd3-40f3-8742-d96cd1736feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:23.876836 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.876821 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl" pod "361af7e8-ecd3-40f3-8742-d96cd1736feb" (UID: "361af7e8-ecd3-40f3-8742-d96cd1736feb"). InnerVolumeSpecName "kube-api-access-2dccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:23.976438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.976381 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.976438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.976405 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361af7e8-ecd3-40f3-8742-d96cd1736feb-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.976438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:23.976416 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dccl\" (UniqueName: \"kubernetes.io/projected/361af7e8-ecd3-40f3-8742-d96cd1736feb-kube-api-access-2dccl\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:24.585988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:24.585961 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" Oct 30 22:17:24.585988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:24.585960 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t" event={"ID":"361af7e8-ecd3-40f3-8742-d96cd1736feb","Type":"ContainerDied","Data":"cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb"} Oct 30 22:17:24.586197 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:24.586017 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb49a0977e039da2deccb9789642f4fe200f8d8fd6338536483b35846d1a37fb" Oct 30 22:17:33.055509 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:33.055466 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd"] Oct 30 22:17:33.106464 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:33.106435 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9hjrd"] Oct 30 22:17:33.428733 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:33.428678 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0818d6dc-1cab-44b4-8c57-5f6bc82429ff" path="/var/lib/kubelet/pods/0818d6dc-1cab-44b4-8c57-5f6bc82429ff/volumes" Oct 30 22:17:44.473298 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:44.473268 4918 scope.go:122] "RemoveContainer" containerID="659f3e2208646ceaf17a3bd223542834464e5bc4cf71995712cd4e91e12b7ec2" Oct 30 22:17:44.484793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:44.484773 4918 scope.go:122] "RemoveContainer" containerID="24c7f4c39432352419aa76d699805910a9cb522d3737a684016f3ca9f91f81ae" Oct 30 22:17:45.801537 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:17:45.801500 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:17:49.402653 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.402617 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn"] Oct 30 22:17:49.403136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.403069 4918 cpu_manager.go:414] "containerMap: removing container" podUID="361af7e8-ecd3-40f3-8742-d96cd1736feb" containerName="revoke-keys" Oct 30 22:17:49.403136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.403080 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361af7e8-ecd3-40f3-8742-d96cd1736feb" containerName="revoke-keys" Oct 30 22:17:49.403210 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.403163 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="361af7e8-ecd3-40f3-8742-d96cd1736feb" containerName="revoke-keys" Oct 30 22:17:49.406358 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.406343 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.441820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.441788 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.441956 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.441832 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktx2\" (UniqueName: \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.442050 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.442032 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.462259 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.462235 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn"] Oct 30 22:17:49.510625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.510603 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:49.510780 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.510686 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:17:49.543259 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.543236 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.543367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.543272 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.543367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.543298 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktx2\" (UniqueName: \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.543868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.543850 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.545668 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.545643 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.605687 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.605666 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktx2\" (UniqueName: \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\") pod \"maas-api-revoke-sub-d2bc93bc-thrvn\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:49.718968 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:49.718915 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:50.095159 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:17:50.095128 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075d195f_a88f_4040_bf19_4a0beea1f0f7.slice/crio-22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798 WatchSource:0}: Error finding container 22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798: Status 404 returned error can't find the container with id 22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798 Oct 30 22:17:50.098357 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:50.098334 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn"] Oct 30 22:17:50.766923 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:50.766833 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" event={"ID":"075d195f-a88f-4040-bf19-4a0beea1f0f7","Type":"ContainerStarted","Data":"0adbdeb16d49d1b4e9737da9cd7b22353cabe4c8011a51b15f1ea450eb227a90"} Oct 30 22:17:50.766923 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:50.766898 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" event={"ID":"075d195f-a88f-4040-bf19-4a0beea1f0f7","Type":"ContainerStarted","Data":"22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798"} Oct 30 22:17:50.887614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:50.887568 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" podStartSLOduration=1.887553287 podStartE2EDuration="1.887553287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:50.886049137 +0000 UTC m=+3851.579531357" watchObservedRunningTime="2026-10-30 22:17:50.887553287 +0000 UTC m=+3851.581035523" Oct 30 22:17:51.624676 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:51.624643 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:51.774584 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:51.774550 4918 generic.go:356] "Generic (PLEG): container finished" podID="075d195f-a88f-4040-bf19-4a0beea1f0f7" containerID="0adbdeb16d49d1b4e9737da9cd7b22353cabe4c8011a51b15f1ea450eb227a90" exitCode=0 Oct 30 22:17:51.775003 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:51.774642 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" event={"ID":"075d195f-a88f-4040-bf19-4a0beea1f0f7","Type":"ContainerDied","Data":"0adbdeb16d49d1b4e9737da9cd7b22353cabe4c8011a51b15f1ea450eb227a90"} Oct 30 22:17:52.939388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:52.939364 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:53.076947 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.076887 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\" (UniqueName: \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\") pod \"075d195f-a88f-4040-bf19-4a0beea1f0f7\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " Oct 30 22:17:53.077069 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.076987 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\") pod \"075d195f-a88f-4040-bf19-4a0beea1f0f7\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " Oct 30 22:17:53.077069 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.077015 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\") pod \"075d195f-a88f-4040-bf19-4a0beea1f0f7\" (UID: \"075d195f-a88f-4040-bf19-4a0beea1f0f7\") " Oct 30 22:17:53.077415 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.077391 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca" pod "075d195f-a88f-4040-bf19-4a0beea1f0f7" (UID: "075d195f-a88f-4040-bf19-4a0beea1f0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:53.078951 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.078929 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2" pod "075d195f-a88f-4040-bf19-4a0beea1f0f7" (UID: "075d195f-a88f-4040-bf19-4a0beea1f0f7"). InnerVolumeSpecName "kube-api-access-gktx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:53.079136 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.079110 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca" pod "075d195f-a88f-4040-bf19-4a0beea1f0f7" (UID: "075d195f-a88f-4040-bf19-4a0beea1f0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:53.178557 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.178533 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktx2\" (UniqueName: \"kubernetes.io/projected/075d195f-a88f-4040-bf19-4a0beea1f0f7-kube-api-access-gktx2\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:53.178557 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.178556 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:53.178688 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.178566 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075d195f-a88f-4040-bf19-4a0beea1f0f7-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:53.788882 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.788811 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" Oct 30 22:17:53.789013 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.788814 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn" event={"ID":"075d195f-a88f-4040-bf19-4a0beea1f0f7","Type":"ContainerDied","Data":"22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798"} Oct 30 22:17:53.789013 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:17:53.788938 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22239f2a1bd032b123d653efeda2c49195f5e1d197c2e0440a58bdd64c418798" Oct 30 22:18:02.053561 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:02.053530 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:02.703309 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:02.703265 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:02.962900 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:02.962814 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:06.958324 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:06.958288 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:07.177442 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:07.177405 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:10.364195 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:10.364157 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:10.555916 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:10.555883 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:13.333199 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.333162 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:18:13.333640 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.333627 4918 cpu_manager.go:414] "containerMap: removing container" podUID="075d195f-a88f-4040-bf19-4a0beea1f0f7" containerName="revoke-keys" Oct 30 22:18:13.333694 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.333641 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075d195f-a88f-4040-bf19-4a0beea1f0f7" containerName="revoke-keys" Oct 30 22:18:13.333769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.333759 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="075d195f-a88f-4040-bf19-4a0beea1f0f7" containerName="revoke-keys" Oct 30 22:18:13.338227 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.338207 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.420973 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.420938 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3ec071-serving-cert\"" Oct 30 22:18:13.456167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.456133 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hkklw\"" Oct 30 22:18:13.456324 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.456218 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:18:13.459602 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.459580 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:18:13.478179 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.478143 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.478335 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.478194 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.478335 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.478282 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6p5\" (UniqueName: \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.478418 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.478403 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.579958 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.579917 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.580175 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.579975 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.580175 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.580005 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6p5\" (UniqueName: \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.580175 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.580071 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.580850 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.580826 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.582625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.582599 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.582736 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.582653 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.586240 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.586178 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:13.642578 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.642538 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6p5\" (UniqueName: \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\") pod \"maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.655579 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.655547 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:13.794617 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:13.794580 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:14.029769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:14.029741 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:18:14.031178 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:18:14.031148 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37d5608_b106_4515_a8a1_d3e686324e54.slice/crio-2f4ebed215d2997109425172f256f43a4bd1fe2710e635b58aa3c37b56dbba13 WatchSource:0}: Error finding container 2f4ebed215d2997109425172f256f43a4bd1fe2710e635b58aa3c37b56dbba13: Status 404 returned error can't find the container with id 2f4ebed215d2997109425172f256f43a4bd1fe2710e635b58aa3c37b56dbba13 Oct 30 22:18:14.921581 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:14.921546 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" event={"ID":"a37d5608-b106-4515-a8a1-d3e686324e54","Type":"ContainerStarted","Data":"2f4ebed215d2997109425172f256f43a4bd1fe2710e635b58aa3c37b56dbba13"} Oct 30 22:18:15.802613 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:18:15.802576 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:18:15.929482 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:15.929438 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" event={"ID":"a37d5608-b106-4515-a8a1-d3e686324e54","Type":"ContainerStarted","Data":"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed"} Oct 30 22:18:15.929979 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:15.929570 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:16.045670 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:16.045605 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" podStartSLOduration=2.2096182 podStartE2EDuration="3.045590698s" totalImagesPullingTime="835.972498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:16.043825825 +0000 UTC m=+3876.737308055" watchObservedRunningTime="2026-10-30 22:18:16.045590698 +0000 UTC m=+3876.739072979" Oct 30 22:18:18.763826 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:18.763781 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:21.941007 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:21.940972 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:18:23.981478 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:23.981443 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:24.234422 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:24.234338 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:29.130883 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:29.130848 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:31.858448 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:31.858411 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:32.496531 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:32.496488 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:32.731477 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:32.731443 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:34.627266 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:34.627233 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:35.790911 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.790872 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:18:35.795781 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.795759 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.895446 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.895413 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ec071\"/\"default-dockercfg-2bwvh\"" Oct 30 22:18:35.895750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.895480 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ec071\"/\"kube-root-ca.crt\"" Oct 30 22:18:35.895750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.895701 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ec071\"/\"simulator-single-main-3ec071-kserve-self-signed-certs\"" Oct 30 22:18:35.895750 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.895701 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ec071\"/\"openshift-service-ca.crt\"" Oct 30 22:18:35.898820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.898802 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:18:35.906378 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906357 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.906432 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906402 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.906493 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906477 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.906572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906516 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8jq\" (UniqueName: \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.906572 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906541 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.906685 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.906654 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:35.982393 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:35.982360 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:36.007624 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007597 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.007777 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007646 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.007855 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007765 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.007855 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007827 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8jq\" (UniqueName: \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.007963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007860 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.007963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.007946 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.008072 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.008039 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.008147 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.008132 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.008318 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.008299 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.010105 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.010088 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.010358 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.010341 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.068732 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.068663 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8jq\" (UniqueName: \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\") pod \"simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.108397 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.108366 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:36.496522 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:36.496497 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:18:36.498019 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:18:36.497986 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd4ea18_3d64_4d04_9483_2448ce2282cb.slice/crio-970074a8a60341feded0ed6d5d2533e40f51ebc58a919c4caa7c25b05684ad93 WatchSource:0}: Error finding container 970074a8a60341feded0ed6d5d2533e40f51ebc58a919c4caa7c25b05684ad93: Status 404 returned error can't find the container with id 970074a8a60341feded0ed6d5d2533e40f51ebc58a919c4caa7c25b05684ad93 Oct 30 22:18:37.064470 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:37.064430 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" event={"ID":"ccd4ea18-3d64-4d04-9483-2448ce2282cb","Type":"ContainerStarted","Data":"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f"} Oct 30 22:18:37.064862 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:37.064481 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" event={"ID":"ccd4ea18-3d64-4d04-9483-2448ce2282cb","Type":"ContainerStarted","Data":"970074a8a60341feded0ed6d5d2533e40f51ebc58a919c4caa7c25b05684ad93"} Oct 30 22:18:37.108625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:37.108599 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:37.186190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:37.186141 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" podStartSLOduration=2.186129445 podStartE2EDuration="2.186129445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:37.183045929 +0000 UTC m=+3897.876528165" watchObservedRunningTime="2026-10-30 22:18:37.186129445 +0000 UTC m=+3897.879611680" Oct 30 22:18:39.593666 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:39.593635 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:40.061325 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.061300 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:18:40.073413 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.073383 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:18:40.074838 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.074812 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:18:40.080636 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.080615 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:18:40.088152 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.088130 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:18:40.093679 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:40.093660 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:18:44.511033 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:44.510997 4918 scope.go:122] "RemoveContainer" containerID="8c286a4bdd207f515b1a47ac96acd709fd9aef4ab0dd09a7a32bd2628520e758" Oct 30 22:18:44.520306 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:44.520286 4918 scope.go:122] "RemoveContainer" containerID="daf2cf7075583fdd42167d7a619398b02576c57b15598771efdd3fd987d143b4" Oct 30 22:18:44.749125 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:44.749084 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:45.821901 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:18:45.821863 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:18:46.108671 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:46.108567 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:46.122220 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:46.122190 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:46.134092 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:46.134056 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:18:49.706324 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:49.706242 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:51.851016 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:51.850985 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:54.724886 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:54.724851 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:59.583938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:18:59.583903 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.589140 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:04.589109 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:07.412867 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:07.412830 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:07.626554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:07.626520 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:09.725612 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:09.725579 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:14.691273 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:14.691238 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:15.802495 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:19:15.802455 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:19:19.664050 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:19.664007 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:24.683501 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:24.683467 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:26.956977 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:26.956926 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:29.980970 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:29.980936 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:31.852800 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:31.852766 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:19:31.981137 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:31.981097 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:35.027303 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:35.027262 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:39.465144 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:39.465113 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:39.655791 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:39.655749 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:39.976054 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:39.976012 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:44.155167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:44.155133 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:19:44.155676 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:44.155363 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" podUID="a37d5608-b106-4515-a8a1-d3e686324e54" containerName="maas-api" containerID="cri-o://7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed" gracePeriod=30 Oct 30 22:19:44.312074 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:44.312036 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:44.548804 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:44.548696 4918 scope.go:122] "RemoveContainer" containerID="11244aa717243655b86ea47e2dc837ffc031349cc3d040c18920cfd1d2b69d1a" Oct 30 22:19:45.423622 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.423598 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:19:45.463604 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.463577 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\") pod \"a37d5608-b106-4515-a8a1-d3e686324e54\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " Oct 30 22:19:45.463766 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.463642 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\" (UniqueName: \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\") pod \"a37d5608-b106-4515-a8a1-d3e686324e54\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " Oct 30 22:19:45.463766 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.463676 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\") pod \"a37d5608-b106-4515-a8a1-d3e686324e54\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " Oct 30 22:19:45.463766 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.463700 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\") pod \"a37d5608-b106-4515-a8a1-d3e686324e54\" (UID: \"a37d5608-b106-4515-a8a1-d3e686324e54\") " Oct 30 22:19:45.464093 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.464066 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca" pod "a37d5608-b106-4515-a8a1-d3e686324e54" (UID: "a37d5608-b106-4515-a8a1-d3e686324e54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:45.465854 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.465819 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5" pod "a37d5608-b106-4515-a8a1-d3e686324e54" (UID: "a37d5608-b106-4515-a8a1-d3e686324e54"). InnerVolumeSpecName "kube-api-access-kq6p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:45.465854 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.465818 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs" pod "a37d5608-b106-4515-a8a1-d3e686324e54" (UID: "a37d5608-b106-4515-a8a1-d3e686324e54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:45.466000 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.465920 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls" pod "a37d5608-b106-4515-a8a1-d3e686324e54" (UID: "a37d5608-b106-4515-a8a1-d3e686324e54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:45.482225 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.482174 4918 generic.go:356] "Generic (PLEG): container finished" podID="a37d5608-b106-4515-a8a1-d3e686324e54" containerID="7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed" exitCode=0 Oct 30 22:19:45.482319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.482229 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" Oct 30 22:19:45.482319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.482254 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" event={"ID":"a37d5608-b106-4515-a8a1-d3e686324e54","Type":"ContainerDied","Data":"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed"} Oct 30 22:19:45.482399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.482326 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5" event={"ID":"a37d5608-b106-4515-a8a1-d3e686324e54","Type":"ContainerDied","Data":"2f4ebed215d2997109425172f256f43a4bd1fe2710e635b58aa3c37b56dbba13"} Oct 30 22:19:45.482399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.482349 4918 scope.go:122] "RemoveContainer" containerID="7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed" Oct 30 22:19:45.492367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.492349 4918 scope.go:122] "RemoveContainer" containerID="7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed" Oct 30 22:19:45.492625 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:19:45.492603 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed\": container with ID starting with 7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed not found: ID does not exist" containerID="7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed" Oct 30 22:19:45.492733 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.492629 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed"} err="failed to get container status \"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed\": rpc error: code = NotFound desc = could not find container \"7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed\": container with ID starting with 7937e7aa7d8c09e5325e89d7b88aead270e90ba53d4323f7a663a384134a04ed not found: ID does not exist" Oct 30 22:19:45.564927 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.564898 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:45.565049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.564931 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6p5\" (UniqueName: \"kubernetes.io/projected/a37d5608-b106-4515-a8a1-d3e686324e54-kube-api-access-kq6p5\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:45.565049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.564947 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37d5608-b106-4515-a8a1-d3e686324e54-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:45.565049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.564961 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37d5608-b106-4515-a8a1-d3e686324e54-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:45.641287 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.641258 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:19:45.646741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.646699 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ec071-7d8f78d757-hhhl5"] Oct 30 22:19:45.802957 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:19:45.802862 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:19:45.856423 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:45.856389 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:46.146422 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:46.146388 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:46.873205 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:46.873146 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerName="main" containerID="cri-o://b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f" gracePeriod=45 Oct 30 22:19:46.909198 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:46.909164 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:47.137014 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.136944 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:19:47.184202 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184162 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184235 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184297 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184364 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184395 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184644 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184447 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\" (UniqueName: \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\") pod \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\" (UID: \"ccd4ea18-3d64-4d04-9483-2448ce2282cb\") " Oct 30 22:19:47.184644 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184486 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.184644 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184513 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.184821 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184717 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.184884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184816 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.184884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184837 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.184884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.184853 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.186874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.186840 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "kube-api-access-rk8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:47.186874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.186859 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:47.186874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.186850 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm" pod "ccd4ea18-3d64-4d04-9483-2448ce2282cb" (UID: "ccd4ea18-3d64-4d04-9483-2448ce2282cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.285872 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.285819 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4ea18-3d64-4d04-9483-2448ce2282cb-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.286063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.285879 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8jq\" (UniqueName: \"kubernetes.io/projected/ccd4ea18-3d64-4d04-9483-2448ce2282cb-kube-api-access-rk8jq\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.286063 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.285897 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4ea18-3d64-4d04-9483-2448ce2282cb-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.428120 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.428038 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37d5608-b106-4515-a8a1-d3e686324e54" path="/var/lib/kubelet/pods/a37d5608-b106-4515-a8a1-d3e686324e54/volumes" Oct 30 22:19:47.497431 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.497398 4918 generic.go:356] "Generic (PLEG): container finished" podID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerID="b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f" exitCode=0 Oct 30 22:19:47.497634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.497438 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" Oct 30 22:19:47.497634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.497479 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" event={"ID":"ccd4ea18-3d64-4d04-9483-2448ce2282cb","Type":"ContainerDied","Data":"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f"} Oct 30 22:19:47.497634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.497530 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" event={"ID":"ccd4ea18-3d64-4d04-9483-2448ce2282cb","Type":"ContainerDied","Data":"970074a8a60341feded0ed6d5d2533e40f51ebc58a919c4caa7c25b05684ad93"} Oct 30 22:19:47.497634 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.497554 4918 scope.go:122] "RemoveContainer" containerID="b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f" Oct 30 22:19:47.507216 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.507192 4918 scope.go:122] "RemoveContainer" containerID="b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f" Oct 30 22:19:47.507515 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:19:47.507495 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f\": container with ID starting with b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f not found: ID does not exist" containerID="b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f" Oct 30 22:19:47.507564 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.507524 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f"} err="failed to get container status \"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f\": rpc error: code = NotFound desc = could not find container \"b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f\": container with ID starting with b40da01fcdafc84e7c10673fcc8b3e8c4f8cec3c21ccaec59ab2537efcacb64f not found: ID does not exist" Oct 30 22:19:47.655228 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.655187 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:19:47.659197 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:47.659166 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp"] Oct 30 22:19:48.124037 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:48.123984 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-3ec071/simulator-single-main-3ec071-kserve-7d5f5777d9-cmqzp" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:49.435975 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:49.435942 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" path="/var/lib/kubelet/pods/ccd4ea18-3d64-4d04-9483-2448ce2282cb/volumes" Oct 30 22:19:52.305840 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:52.305801 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:52.506229 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:52.506201 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.186977 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:53.186941 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.434461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:53.434427 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.643040 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:53.643011 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:59.959107 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959070 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:19:59.959614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959520 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerName="main" Oct 30 22:19:59.959614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959531 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerName="main" Oct 30 22:19:59.959614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959557 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a37d5608-b106-4515-a8a1-d3e686324e54" containerName="maas-api" Oct 30 22:19:59.959614 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959563 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37d5608-b106-4515-a8a1-d3e686324e54" containerName="maas-api" Oct 30 22:19:59.959820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959626 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4ea18-3d64-4d04-9483-2448ce2282cb" containerName="main" Oct 30 22:19:59.959820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.959642 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37d5608-b106-4515-a8a1-d3e686324e54" containerName="maas-api" Oct 30 22:19:59.964335 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:19:59.964316 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.000769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.000728 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmwl\" (UniqueName: \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.000910 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.000793 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.000910 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.000845 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.001030 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.000912 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.001030 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.000946 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.001149 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.001104 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.066877 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.066851 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-38871958-kserve-self-signed-certs\"" Oct 30 22:20:00.069795 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.069770 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:20:00.102317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102116 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmwl\" (UniqueName: \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.102317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102171 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.102317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102214 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.102317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102242 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.102317 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102269 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.102692 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.102349 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.103041 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.103018 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.103771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.103743 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.107211 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.107162 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.109547 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.109523 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.111593 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.111568 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.114276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.114246 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.160222 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.160197 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmwl\" (UniqueName: \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\") pod \"e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.277552 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.277486 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:00.648521 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:00.648493 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:20:00.649344 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:20:00.649315 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc42f5f90_c7fd_44e4_a43a_0baf4ff314a1.slice/crio-29a3f72c695cf1cde9cd1663a9cefad97b0da1681a55e328e7d32649fd5e0342 WatchSource:0}: Error finding container 29a3f72c695cf1cde9cd1663a9cefad97b0da1681a55e328e7d32649fd5e0342: Status 404 returned error can't find the container with id 29a3f72c695cf1cde9cd1663a9cefad97b0da1681a55e328e7d32649fd5e0342 Oct 30 22:20:01.581093 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:01.581052 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" event={"ID":"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1","Type":"ContainerStarted","Data":"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51"} Oct 30 22:20:01.581093 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:01.581096 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" event={"ID":"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1","Type":"ContainerStarted","Data":"29a3f72c695cf1cde9cd1663a9cefad97b0da1681a55e328e7d32649fd5e0342"} Oct 30 22:20:01.694540 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:01.694485 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" podStartSLOduration=2.694470229 podStartE2EDuration="2.694470229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:01.691383637 +0000 UTC m=+3982.384865874" watchObservedRunningTime="2026-10-30 22:20:01.694470229 +0000 UTC m=+3982.387952515" Oct 30 22:20:02.278196 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:02.278165 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:08.469618 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:08.469585 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:09.967684 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:09.967639 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:10.274154 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:10.274076 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:10.277671 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:10.277644 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:10.290108 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:10.290090 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:10.486682 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:10.486655 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:10.689893 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:10.689866 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:20:15.803412 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:20:15.803376 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:20:20.054641 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:20.054554 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv"] Oct 30 22:20:20.100101 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:20.100064 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsnkv"] Oct 30 22:20:21.427477 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:21.427440 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168ecedc-1940-4610-b18a-a32a80762d36" path="/var/lib/kubelet/pods/168ecedc-1940-4610-b18a-a32a80762d36/volumes" Oct 30 22:20:25.954086 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:25.954052 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:43.087030 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:43.086996 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:45.800921 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:20:45.800884 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:20:52.507793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:20:52.507763 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:15.801142 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:21:15.801105 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:21:17.415168 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.415133 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh"] Oct 30 22:21:17.418854 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.418833 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.517587 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.517554 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:21:17.517738 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.517584 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:21:17.520494 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.520464 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh"] Oct 30 22:21:17.526343 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.526315 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvmx\" (UniqueName: \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.526453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.526347 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.526453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.526379 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.627921 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.627889 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvmx\" (UniqueName: \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.627921 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.627924 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.628105 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.627947 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.628665 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.628647 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.630350 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.630332 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.685427 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.685367 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvmx\" (UniqueName: \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\") pod \"maas-api-revoke-sub-452b6d7b-7mvvh\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:17.733166 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:17.733137 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:18.106916 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:18.106883 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh"] Oct 30 22:21:18.109883 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:21:18.109834 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c009168_e40e_4001_8add_7e3f01f448cc.slice/crio-81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad WatchSource:0}: Error finding container 81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad: Status 404 returned error can't find the container with id 81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad Oct 30 22:21:18.124083 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:18.124053 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" event={"ID":"3c009168-e40e-4001-8add-7e3f01f448cc","Type":"ContainerStarted","Data":"81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad"} Oct 30 22:21:19.131331 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:19.131295 4918 generic.go:356] "Generic (PLEG): container finished" podID="3c009168-e40e-4001-8add-7e3f01f448cc" containerID="8071c37108735848db425a0e5c22c8d2cc110d642b6fd2257b98070d63d65bbe" exitCode=0 Oct 30 22:21:19.131667 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:19.131378 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" event={"ID":"3c009168-e40e-4001-8add-7e3f01f448cc","Type":"ContainerDied","Data":"8071c37108735848db425a0e5c22c8d2cc110d642b6fd2257b98070d63d65bbe"} Oct 30 22:21:20.325591 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.325286 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:20.456288 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.455411 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\") pod \"3c009168-e40e-4001-8add-7e3f01f448cc\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " Oct 30 22:21:20.456288 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.455540 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\" (UniqueName: \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\") pod \"3c009168-e40e-4001-8add-7e3f01f448cc\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " Oct 30 22:21:20.456288 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.455592 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\") pod \"3c009168-e40e-4001-8add-7e3f01f448cc\" (UID: \"3c009168-e40e-4001-8add-7e3f01f448cc\") " Oct 30 22:21:20.456876 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.456834 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca" pod "3c009168-e40e-4001-8add-7e3f01f448cc" (UID: "3c009168-e40e-4001-8add-7e3f01f448cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:20.458748 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.458676 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca" pod "3c009168-e40e-4001-8add-7e3f01f448cc" (UID: "3c009168-e40e-4001-8add-7e3f01f448cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:20.458858 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.458737 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx" pod "3c009168-e40e-4001-8add-7e3f01f448cc" (UID: "3c009168-e40e-4001-8add-7e3f01f448cc"). InnerVolumeSpecName "kube-api-access-plvmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:20.558405 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.557567 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:20.558405 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.557599 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvmx\" (UniqueName: \"kubernetes.io/projected/3c009168-e40e-4001-8add-7e3f01f448cc-kube-api-access-plvmx\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:20.558405 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:20.557614 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c009168-e40e-4001-8add-7e3f01f448cc-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:21.155190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:21.155147 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" event={"ID":"3c009168-e40e-4001-8add-7e3f01f448cc","Type":"ContainerDied","Data":"81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad"} Oct 30 22:21:21.155190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:21.155188 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh" Oct 30 22:21:21.155416 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:21.155211 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f034cf170b3351267f8d7792d5e356a65a0edc895a72bcea3b9827e699c8ad" Oct 30 22:21:30.338617 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:30.338578 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:21:30.436559 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:30.436531 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:44.594687 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:44.594645 4918 scope.go:122] "RemoveContainer" containerID="cf92504199dd218a6de5a8aed3eb09dbaabed911b5913c0ee07978575e02e6aa" Oct 30 22:21:45.358611 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.358556 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" podUID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" containerName="main" containerID="cri-o://62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51" gracePeriod=45 Oct 30 22:21:45.618397 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.618371 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:21:45.619385 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619364 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619455 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619439 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619509 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619461 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\" (UniqueName: \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619509 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619484 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619620 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619524 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619620 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619577 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\") pod \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\" (UID: \"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1\") " Oct 30 22:21:45.619815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619787 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:45.619898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619803 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:45.619980 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619936 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-model-cache\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.619980 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619935 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:45.619980 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.619953 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-home\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.621587 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.621559 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:45.621699 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.621657 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "kube-api-access-bjmwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:45.621699 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.621664 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs" pod "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" (UID: "c42f5f90-c7fd-44e4-a43a-0baf4ff314a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:45.720734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.720690 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tls-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.720734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.720730 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-dshm\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.720928 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.720741 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmwl\" (UniqueName: \"kubernetes.io/projected/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-kube-api-access-bjmwl\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.720928 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:45.720752 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1-tmp-dir\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.801752 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:21:45.801691 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:21:46.314412 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.314379 4918 generic.go:356] "Generic (PLEG): container finished" podID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" containerID="62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51" exitCode=0 Oct 30 22:21:46.314585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.314419 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" Oct 30 22:21:46.314585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.314462 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" event={"ID":"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1","Type":"ContainerDied","Data":"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51"} Oct 30 22:21:46.314585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.314504 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn" event={"ID":"c42f5f90-c7fd-44e4-a43a-0baf4ff314a1","Type":"ContainerDied","Data":"29a3f72c695cf1cde9cd1663a9cefad97b0da1681a55e328e7d32649fd5e0342"} Oct 30 22:21:46.314585 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.314523 4918 scope.go:122] "RemoveContainer" containerID="62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51" Oct 30 22:21:46.327860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.327841 4918 scope.go:122] "RemoveContainer" containerID="62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51" Oct 30 22:21:46.328163 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:21:46.328141 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51\": container with ID starting with 62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51 not found: ID does not exist" containerID="62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51" Oct 30 22:21:46.328238 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.328173 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51"} err="failed to get container status \"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51\": rpc error: code = NotFound desc = could not find container \"62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51\": container with ID starting with 62f24f2964fcf357be58bfa14bf2818b4cc1888c74a4550c602bb4e3812ddf51 not found: ID does not exist" Oct 30 22:21:46.470052 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.470025 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:21:46.516543 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:46.516517 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-38871958-kserve-7c7fc5d49c-4dzmn"] Oct 30 22:21:47.426866 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:47.426836 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" path="/var/lib/kubelet/pods/c42f5f90-c7fd-44e4-a43a-0baf4ff314a1/volumes" Oct 30 22:21:49.020870 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:49.020831 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:52.922247 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:52.922211 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:53.159668 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.159636 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc"] Oct 30 22:21:53.160225 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160201 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3c009168-e40e-4001-8add-7e3f01f448cc" containerName="revoke-keys" Oct 30 22:21:53.160225 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160224 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c009168-e40e-4001-8add-7e3f01f448cc" containerName="revoke-keys" Oct 30 22:21:53.160419 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160279 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" containerName="main" Oct 30 22:21:53.160419 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160287 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" containerName="main" Oct 30 22:21:53.160528 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160425 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42f5f90-c7fd-44e4-a43a-0baf4ff314a1" containerName="main" Oct 30 22:21:53.160528 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.160445 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c009168-e40e-4001-8add-7e3f01f448cc" containerName="revoke-keys" Oct 30 22:21:53.163805 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.163784 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.186135 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.186064 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdpk\" (UniqueName: \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.186353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.186316 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.186481 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.186442 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.214801 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.214777 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:21:53.214949 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.214815 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:21:53.222216 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.222190 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc"] Oct 30 22:21:53.287173 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.287141 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.287319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.287284 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdpk\" (UniqueName: \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.287367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.287322 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.287976 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.287958 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.290018 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.289993 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.323936 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.323912 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:53.345390 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.345367 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdpk\" (UniqueName: \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\") pod \"maas-api-revoke-sub-3e31a506-fwtwc\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.478042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.477981 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:53.705565 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.705527 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:53.846884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:53.846853 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc"] Oct 30 22:21:53.847552 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:21:53.847496 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63b9089_1f47_41e4_a0c4_5c381e17e050.slice/crio-9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435 WatchSource:0}: Error finding container 9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435: Status 404 returned error can't find the container with id 9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435 Oct 30 22:21:54.367324 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:54.367288 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" event={"ID":"d63b9089-1f47-41e4-a0c4-5c381e17e050","Type":"ContainerStarted","Data":"ba0518c014e7db45df8ef0ef3989e7179b99505a78809756f6cbfddae314574e"} Oct 30 22:21:54.367753 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:54.367358 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" event={"ID":"d63b9089-1f47-41e4-a0c4-5c381e17e050","Type":"ContainerStarted","Data":"9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435"} Oct 30 22:21:54.484554 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:54.484444 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" podStartSLOduration=1.484425971 podStartE2EDuration="1.484425971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:54.480333655 +0000 UTC m=+4095.173815893" watchObservedRunningTime="2026-10-30 22:21:54.484425971 +0000 UTC m=+4095.177908207" Oct 30 22:21:55.422729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:55.419762 4918 generic.go:356] "Generic (PLEG): container finished" podID="d63b9089-1f47-41e4-a0c4-5c381e17e050" containerID="ba0518c014e7db45df8ef0ef3989e7179b99505a78809756f6cbfddae314574e" exitCode=0 Oct 30 22:21:55.422729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:55.420011 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" event={"ID":"d63b9089-1f47-41e4-a0c4-5c381e17e050","Type":"ContainerDied","Data":"ba0518c014e7db45df8ef0ef3989e7179b99505a78809756f6cbfddae314574e"} Oct 30 22:21:56.589541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.589515 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:56.659182 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.659158 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\") pod \"d63b9089-1f47-41e4-a0c4-5c381e17e050\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " Oct 30 22:21:56.659327 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.659253 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\") pod \"d63b9089-1f47-41e4-a0c4-5c381e17e050\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " Oct 30 22:21:56.659327 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.659272 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\" (UniqueName: \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\") pod \"d63b9089-1f47-41e4-a0c4-5c381e17e050\" (UID: \"d63b9089-1f47-41e4-a0c4-5c381e17e050\") " Oct 30 22:21:56.659696 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.659663 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca" pod "d63b9089-1f47-41e4-a0c4-5c381e17e050" (UID: "d63b9089-1f47-41e4-a0c4-5c381e17e050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:56.661190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.661163 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca" pod "d63b9089-1f47-41e4-a0c4-5c381e17e050" (UID: "d63b9089-1f47-41e4-a0c4-5c381e17e050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:56.661396 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.661374 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk" pod "d63b9089-1f47-41e4-a0c4-5c381e17e050" (UID: "d63b9089-1f47-41e4-a0c4-5c381e17e050"). InnerVolumeSpecName "kube-api-access-zsdpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:56.760751 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.760656 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:56.760751 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.760733 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63b9089-1f47-41e4-a0c4-5c381e17e050-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:56.760937 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:56.760750 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdpk\" (UniqueName: \"kubernetes.io/projected/d63b9089-1f47-41e4-a0c4-5c381e17e050-kube-api-access-zsdpk\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:57.434128 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:57.434095 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" Oct 30 22:21:57.434292 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:57.434099 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc" event={"ID":"d63b9089-1f47-41e4-a0c4-5c381e17e050","Type":"ContainerDied","Data":"9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435"} Oct 30 22:21:57.434292 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:21:57.434262 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9042ff8f2c6e8b41388a00945fe79188e637059c9a9773a620dd88f36375a435" Oct 30 22:22:15.801060 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:22:15.801026 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:22:24.050864 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:24.050830 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t"] Oct 30 22:22:24.096395 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:24.096363 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5jj8t"] Oct 30 22:22:25.427190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:25.427157 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361af7e8-ecd3-40f3-8742-d96cd1736feb" path="/var/lib/kubelet/pods/361af7e8-ecd3-40f3-8742-d96cd1736feb/volumes" Oct 30 22:22:26.999618 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:26.999585 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4"] Oct 30 22:22:27.000093 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.000079 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d63b9089-1f47-41e4-a0c4-5c381e17e050" containerName="revoke-keys" Oct 30 22:22:27.000186 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.000093 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63b9089-1f47-41e4-a0c4-5c381e17e050" containerName="revoke-keys" Oct 30 22:22:27.000186 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.000170 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63b9089-1f47-41e4-a0c4-5c381e17e050" containerName="revoke-keys" Oct 30 22:22:27.003719 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.003689 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.056270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.056252 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:27.080766 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.080741 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.080869 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.080785 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj567\" (UniqueName: \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.080914 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.080882 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.102869 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.102850 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:22:27.105898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.105879 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4"] Oct 30 22:22:27.182241 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.182212 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.182365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.182312 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.182440 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.182369 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj567\" (UniqueName: \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.182918 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.182898 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.184661 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.184638 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.241582 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.241556 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj567\" (UniqueName: \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\") pod \"maas-api-revoke-sub-00985ecb-52cc4\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.316839 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.316787 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:27.701362 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:27.701329 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4"] Oct 30 22:22:27.704296 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:22:27.704267 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6b0769_f882_45f9_8388_4ccd60cdc65a.slice/crio-ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513 WatchSource:0}: Error finding container ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513: Status 404 returned error can't find the container with id ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513 Oct 30 22:22:28.712060 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:28.712026 4918 generic.go:356] "Generic (PLEG): container finished" podID="7a6b0769-f882-45f9-8388-4ccd60cdc65a" containerID="6e436f416b7ad8729425447a6087aa4100a0f9b6e5ea9077dd6e6c8245f9d2fc" exitCode=0 Oct 30 22:22:28.712403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:28.712107 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" event={"ID":"7a6b0769-f882-45f9-8388-4ccd60cdc65a","Type":"ContainerDied","Data":"6e436f416b7ad8729425447a6087aa4100a0f9b6e5ea9077dd6e6c8245f9d2fc"} Oct 30 22:22:28.712403 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:28.712154 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" event={"ID":"7a6b0769-f882-45f9-8388-4ccd60cdc65a","Type":"ContainerStarted","Data":"ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513"} Oct 30 22:22:29.856854 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.856829 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:29.906553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.906524 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\" (UniqueName: \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\") pod \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " Oct 30 22:22:29.906769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.906585 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\") pod \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " Oct 30 22:22:29.906769 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.906675 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\") pod \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\" (UID: \"7a6b0769-f882-45f9-8388-4ccd60cdc65a\") " Oct 30 22:22:29.907065 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.907043 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca" pod "7a6b0769-f882-45f9-8388-4ccd60cdc65a" (UID: "7a6b0769-f882-45f9-8388-4ccd60cdc65a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:29.908674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.908652 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca" pod "7a6b0769-f882-45f9-8388-4ccd60cdc65a" (UID: "7a6b0769-f882-45f9-8388-4ccd60cdc65a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:29.908774 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:29.908672 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567" pod "7a6b0769-f882-45f9-8388-4ccd60cdc65a" (UID: "7a6b0769-f882-45f9-8388-4ccd60cdc65a"). InnerVolumeSpecName "kube-api-access-lj567". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:30.008171 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.008097 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-xks-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:30.008171 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.008127 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6b0769-f882-45f9-8388-4ccd60cdc65a-maas-api-openshift-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:30.008171 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.008141 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj567\" (UniqueName: \"kubernetes.io/projected/7a6b0769-f882-45f9-8388-4ccd60cdc65a-kube-api-access-lj567\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:30.725270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.725228 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" event={"ID":"7a6b0769-f882-45f9-8388-4ccd60cdc65a","Type":"ContainerDied","Data":"ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513"} Oct 30 22:22:30.725270 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.725258 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4" Oct 30 22:22:30.725492 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:30.725312 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee06781cba3417b699519978f201c3aa646349319cf3d000f0d65ee46453d513" Oct 30 22:22:40.449672 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.449641 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:40.450168 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.450153 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7a6b0769-f882-45f9-8388-4ccd60cdc65a" containerName="revoke-keys" Oct 30 22:22:40.450220 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.450169 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6b0769-f882-45f9-8388-4ccd60cdc65a" containerName="revoke-keys" Oct 30 22:22:40.450256 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.450250 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6b0769-f882-45f9-8388-4ccd60cdc65a" containerName="revoke-keys" Oct 30 22:22:40.453796 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.453770 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.465318 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.465294 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6687d569c9-scd7k"] Oct 30 22:22:40.469058 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.469039 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.508814 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.508792 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:22:40.549648 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.549625 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hkklw\"" Oct 30 22:22:40.550119 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.550087 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:22:40.554154 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.554125 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:40.556831 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.556809 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6687d569c9-scd7k"] Oct 30 22:22:40.597528 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597497 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.597647 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597542 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntqg\" (UniqueName: \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.597734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597644 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-metrics-certs\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.597734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597667 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.597734 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597686 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.597911 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597747 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24911709-2140-48d5-9a31-8159c398ebf5-openshift-service-ca\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.597911 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597884 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8fx\" (UniqueName: \"kubernetes.io/projected/24911709-2140-48d5-9a31-8159c398ebf5-kube-api-access-bh8fx\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.598020 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.597912 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-maas-api-tls\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.698627 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698596 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.698764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698643 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntqg\" (UniqueName: \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.698764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698734 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-metrics-certs\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.698898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698769 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.698898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698798 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.698898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698835 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24911709-2140-48d5-9a31-8159c398ebf5-openshift-service-ca\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.698898 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698888 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8fx\" (UniqueName: \"kubernetes.io/projected/24911709-2140-48d5-9a31-8159c398ebf5-kube-api-access-bh8fx\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.699142 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.698920 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-maas-api-tls\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.699484 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.699461 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.699877 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.699824 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24911709-2140-48d5-9a31-8159c398ebf5-openshift-service-ca\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.701276 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.701250 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-metrics-certs\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.701454 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.701433 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.701699 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.701682 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24911709-2140-48d5-9a31-8159c398ebf5-maas-api-tls\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.701760 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.701733 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.759588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.759570 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntqg\" (UniqueName: \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\") pod \"maas-api-86b5466f75-xxl7k\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.767042 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.767024 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:40.773294 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.773273 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8fx\" (UniqueName: \"kubernetes.io/projected/24911709-2140-48d5-9a31-8159c398ebf5-kube-api-access-bh8fx\") pod \"maas-api-6687d569c9-scd7k\" (UID: \"24911709-2140-48d5-9a31-8159c398ebf5\") " pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:40.781946 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:40.781925 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:41.180957 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:41.180928 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:41.182593 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:22:41.182566 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ef10ae_064b_42b5_b471_322bc9da34bf.slice/crio-4ead7b5bbd18fd3793e9b2e7592e9587d646eb051d8d2a97ff219e31d5019c9e WatchSource:0}: Error finding container 4ead7b5bbd18fd3793e9b2e7592e9587d646eb051d8d2a97ff219e31d5019c9e: Status 404 returned error can't find the container with id 4ead7b5bbd18fd3793e9b2e7592e9587d646eb051d8d2a97ff219e31d5019c9e Oct 30 22:22:41.184082 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:41.184062 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:22:41.195621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:41.195594 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6687d569c9-scd7k"] Oct 30 22:22:41.218879 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:22:41.218857 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24911709_2140_48d5_9a31_8159c398ebf5.slice/crio-d75cedc5ee33aeaf9d305422fcafa6949c56b488f13925fb183cc384a4eca91c WatchSource:0}: Error finding container d75cedc5ee33aeaf9d305422fcafa6949c56b488f13925fb183cc384a4eca91c: Status 404 returned error can't find the container with id d75cedc5ee33aeaf9d305422fcafa6949c56b488f13925fb183cc384a4eca91c Oct 30 22:22:41.799696 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:41.799662 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" event={"ID":"24911709-2140-48d5-9a31-8159c398ebf5","Type":"ContainerStarted","Data":"d75cedc5ee33aeaf9d305422fcafa6949c56b488f13925fb183cc384a4eca91c"} Oct 30 22:22:41.801068 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:41.801045 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" event={"ID":"77ef10ae-064b-42b5-b471-322bc9da34bf","Type":"ContainerStarted","Data":"4ead7b5bbd18fd3793e9b2e7592e9587d646eb051d8d2a97ff219e31d5019c9e"} Oct 30 22:22:42.810035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:42.809984 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" event={"ID":"77ef10ae-064b-42b5-b471-322bc9da34bf","Type":"ContainerStarted","Data":"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b"} Oct 30 22:22:42.810513 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:42.810107 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:42.812201 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:42.812173 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" event={"ID":"24911709-2140-48d5-9a31-8159c398ebf5","Type":"ContainerStarted","Data":"eefa77cb182933f3257b41d64ca50f72a0cc8c0994ff1f2317f0d5ecc6e4acbd"} Oct 30 22:22:42.812331 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:42.812235 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:42.923910 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:42.923860 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" podStartSLOduration=2.028067553 podStartE2EDuration="2.923840568s" totalImagesPullingTime="895.773015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:42.921618932 +0000 UTC m=+4143.615101169" watchObservedRunningTime="2026-10-30 22:22:42.923840568 +0000 UTC m=+4143.617322832" Oct 30 22:22:43.038070 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:43.038016 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" podStartSLOduration=2.178734674 podStartE2EDuration="3.038002999s" totalImagesPullingTime="859.268325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:43.037083824 +0000 UTC m=+4143.730566064" watchObservedRunningTime="2026-10-30 22:22:43.038002999 +0000 UTC m=+4143.731485265" Oct 30 22:22:45.801885 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:22:45.801835 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:22:47.224548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:47.224509 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:47.789676 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:47.789636 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:48.043948 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:48.043865 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:48.824383 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:48.824354 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:48.824874 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:48.824849 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6687d569c9-scd7k" Oct 30 22:22:49.131868 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.131834 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:49.132075 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.132051 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" podUID="77ef10ae-064b-42b5-b471-322bc9da34bf" containerName="maas-api" containerID="cri-o://c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b" gracePeriod=30 Oct 30 22:22:49.385650 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.385581 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:49.486884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.486852 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\") pod \"77ef10ae-064b-42b5-b471-322bc9da34bf\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " Oct 30 22:22:49.487019 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.486944 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\") pod \"77ef10ae-064b-42b5-b471-322bc9da34bf\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " Oct 30 22:22:49.487019 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.487008 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\") pod \"77ef10ae-064b-42b5-b471-322bc9da34bf\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " Oct 30 22:22:49.487135 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.487120 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\" (UniqueName: \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\") pod \"77ef10ae-064b-42b5-b471-322bc9da34bf\" (UID: \"77ef10ae-064b-42b5-b471-322bc9da34bf\") " Oct 30 22:22:49.487443 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.487410 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca" pod "77ef10ae-064b-42b5-b471-322bc9da34bf" (UID: "77ef10ae-064b-42b5-b471-322bc9da34bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:49.489096 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.489065 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls" pod "77ef10ae-064b-42b5-b471-322bc9da34bf" (UID: "77ef10ae-064b-42b5-b471-322bc9da34bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:49.489228 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.489183 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs" pod "77ef10ae-064b-42b5-b471-322bc9da34bf" (UID: "77ef10ae-064b-42b5-b471-322bc9da34bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:49.489593 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.489567 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg" pod "77ef10ae-064b-42b5-b471-322bc9da34bf" (UID: "77ef10ae-064b-42b5-b471-322bc9da34bf"). InnerVolumeSpecName "kube-api-access-tntqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:49.588621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.588591 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntqg\" (UniqueName: \"kubernetes.io/projected/77ef10ae-064b-42b5-b471-322bc9da34bf-kube-api-access-tntqg\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.588621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.588615 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.588621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.588626 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ef10ae-064b-42b5-b471-322bc9da34bf-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.588881 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.588639 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ef10ae-064b-42b5-b471-322bc9da34bf-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.853927 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.853829 4918 generic.go:356] "Generic (PLEG): container finished" podID="77ef10ae-064b-42b5-b471-322bc9da34bf" containerID="c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b" exitCode=0 Oct 30 22:22:49.853927 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.853898 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" Oct 30 22:22:49.854468 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.853916 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" event={"ID":"77ef10ae-064b-42b5-b471-322bc9da34bf","Type":"ContainerDied","Data":"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b"} Oct 30 22:22:49.854468 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.853995 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k" event={"ID":"77ef10ae-064b-42b5-b471-322bc9da34bf","Type":"ContainerDied","Data":"4ead7b5bbd18fd3793e9b2e7592e9587d646eb051d8d2a97ff219e31d5019c9e"} Oct 30 22:22:49.854468 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.854023 4918 scope.go:122] "RemoveContainer" containerID="c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b" Oct 30 22:22:49.865128 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.865108 4918 scope.go:122] "RemoveContainer" containerID="c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b" Oct 30 22:22:49.865364 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:22:49.865346 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b\": container with ID starting with c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b not found: ID does not exist" containerID="c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b" Oct 30 22:22:49.865407 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:49.865373 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b"} err="failed to get container status \"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b\": rpc error: code = NotFound desc = could not find container \"c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b\": container with ID starting with c7148640fac5e4b3fb4b8db9fbba92ca085032b00f5c9901a746bc6f3e77f23b not found: ID does not exist" Oct 30 22:22:50.014142 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:50.014117 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:50.021058 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:50.021036 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5466f75-xxl7k"] Oct 30 22:22:50.959178 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:50.959142 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:51.194062 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:51.194026 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:51.427922 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:51.427888 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ef10ae-064b-42b5-b471-322bc9da34bf" path="/var/lib/kubelet/pods/77ef10ae-064b-42b5-b471-322bc9da34bf/volumes" Oct 30 22:22:53.074962 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:53.074929 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn"] Oct 30 22:22:53.121524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:53.121490 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-thrvn"] Oct 30 22:22:53.204357 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:53.204325 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:53.427880 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:53.427848 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075d195f-a88f-4040-bf19-4a0beea1f0f7" path="/var/lib/kubelet/pods/075d195f-a88f-4040-bf19-4a0beea1f0f7/volumes" Oct 30 22:22:53.490106 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:53.490080 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:54.318353 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:54.318312 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:54.523562 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:54.523531 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.215154 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.215124 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:22:56.215621 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.215608 4918 cpu_manager.go:414] "containerMap: removing container" podUID="77ef10ae-064b-42b5-b471-322bc9da34bf" containerName="maas-api" Oct 30 22:22:56.215677 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.215622 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ef10ae-064b-42b5-b471-322bc9da34bf" containerName="maas-api" Oct 30 22:22:56.215741 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.215697 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ef10ae-064b-42b5-b471-322bc9da34bf" containerName="maas-api" Oct 30 22:22:56.220283 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.220267 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.269637 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.269615 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8c501563-serving-cert\"" Oct 30 22:22:56.276355 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.276290 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:22:56.360219 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.360190 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.360365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.360240 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7mk\" (UniqueName: \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.360365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.360332 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.360506 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.360485 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.461460 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.461430 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.461660 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.461506 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.461660 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.461537 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7mk\" (UniqueName: \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.461660 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.461573 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.461918 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:22:56.461670 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-serving-cert: secret "maas-api-e2e-ait-8c501563-serving-cert" not found Oct 30 22:22:56.461918 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:22:56.461786 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls podName:42245ba5-9d23-479e-aee2-2e5d118c28bc nodeName:}" failed. No retries permitted until 2026-10-30 22:22:56.96176378 +0000 UTC m=+4157.655245997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls") pod "maas-api-e2e-ait-8c501563-568688dc5b-5msrd" (UID: "42245ba5-9d23-479e-aee2-2e5d118c28bc") : secret "maas-api-e2e-ait-8c501563-serving-cert" not found Oct 30 22:22:56.462375 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.462351 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.464671 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.464642 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.469160 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.469076 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.522297 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.522268 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7mk\" (UniqueName: \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.664547 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.664509 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.860514 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.860487 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.966207 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.966177 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:56.968739 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:56.968721 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") pod \"maas-api-e2e-ait-8c501563-568688dc5b-5msrd\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:57.133404 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:57.133344 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:57.505988 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:57.505961 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:22:57.507533 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:22:57.507506 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42245ba5_9d23_479e_aee2_2e5d118c28bc.slice/crio-7d8ff48f4e2058c25c9cc0f65bb5dc8b5b5f934201a2aa2d855bfc9a68a8e776 WatchSource:0}: Error finding container 7d8ff48f4e2058c25c9cc0f65bb5dc8b5b5f934201a2aa2d855bfc9a68a8e776: Status 404 returned error can't find the container with id 7d8ff48f4e2058c25c9cc0f65bb5dc8b5b5f934201a2aa2d855bfc9a68a8e776 Oct 30 22:22:57.905114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:57.905069 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" event={"ID":"42245ba5-9d23-479e-aee2-2e5d118c28bc","Type":"ContainerStarted","Data":"7d8ff48f4e2058c25c9cc0f65bb5dc8b5b5f934201a2aa2d855bfc9a68a8e776"} Oct 30 22:22:58.913583 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:58.913548 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" event={"ID":"42245ba5-9d23-479e-aee2-2e5d118c28bc","Type":"ContainerStarted","Data":"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c"} Oct 30 22:22:58.914070 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:58.913605 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:22:59.027216 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:22:59.027159 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" podStartSLOduration=2.174373662 podStartE2EDuration="3.027146007s" totalImagesPullingTime="852.772345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:59.024446103 +0000 UTC m=+4159.717928339" watchObservedRunningTime="2026-10-30 22:22:59.027146007 +0000 UTC m=+4159.720628306" Oct 30 22:23:01.991469 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:01.991437 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:04.925215 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:04.925186 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:23:07.079725 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:07.079670 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:12.140899 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:12.140868 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:15.801793 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:23:15.801755 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:23:17.645817 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:17.645776 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:22.989250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:22.989164 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:27.969190 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:27.969158 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:32.866660 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:32.866623 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:37.857921 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:37.857888 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.437049 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:39.436971 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.630695 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:39.630657 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:40.107553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.107528 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:23:40.121721 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.121678 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:23:40.121893 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.121862 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:23:40.127143 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.127120 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:23:40.134888 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.134870 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:23:40.139803 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:40.139784 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:23:42.692455 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:42.692422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:44.633327 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:44.633291 4918 scope.go:122] "RemoveContainer" containerID="8c6e42607388209a7e9270aecd48aabbbc74abdb6e5c2440e38929ab7da40892" Oct 30 22:23:45.801106 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:23:45.801070 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:23:46.894209 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:46.894176 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:23:46.894786 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:46.894388 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" podUID="42245ba5-9d23-479e-aee2-2e5d118c28bc" containerName="maas-api" containerID="cri-o://c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c" gracePeriod=30 Oct 30 22:23:47.043843 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.043785 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:47.162795 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.162773 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:23:47.215406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.215376 4918 generic.go:356] "Generic (PLEG): container finished" podID="42245ba5-9d23-479e-aee2-2e5d118c28bc" containerID="c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c" exitCode=0 Oct 30 22:23:47.215524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.215424 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" Oct 30 22:23:47.215524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.215463 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" event={"ID":"42245ba5-9d23-479e-aee2-2e5d118c28bc","Type":"ContainerDied","Data":"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c"} Oct 30 22:23:47.215524 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.215518 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd" event={"ID":"42245ba5-9d23-479e-aee2-2e5d118c28bc","Type":"ContainerDied","Data":"7d8ff48f4e2058c25c9cc0f65bb5dc8b5b5f934201a2aa2d855bfc9a68a8e776"} Oct 30 22:23:47.215659 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.215550 4918 scope.go:122] "RemoveContainer" containerID="c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c" Oct 30 22:23:47.225288 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.225269 4918 scope.go:122] "RemoveContainer" containerID="c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c" Oct 30 22:23:47.225552 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:23:47.225534 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c\": container with ID starting with c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c not found: ID does not exist" containerID="c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c" Oct 30 22:23:47.225610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.225562 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c"} err="failed to get container status \"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c\": rpc error: code = NotFound desc = could not find container \"c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c\": container with ID starting with c00d0eb71a26baa9c21038fa9ce8fa56671480f9350444759a6631d53a330b4c not found: ID does not exist" Oct 30 22:23:47.234539 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.234519 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\") pod \"42245ba5-9d23-479e-aee2-2e5d118c28bc\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " Oct 30 22:23:47.234630 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.234568 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\" (UniqueName: \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\") pod \"42245ba5-9d23-479e-aee2-2e5d118c28bc\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " Oct 30 22:23:47.234630 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.234587 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\") pod \"42245ba5-9d23-479e-aee2-2e5d118c28bc\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " Oct 30 22:23:47.234768 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.234637 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") pod \"42245ba5-9d23-479e-aee2-2e5d118c28bc\" (UID: \"42245ba5-9d23-479e-aee2-2e5d118c28bc\") " Oct 30 22:23:47.235046 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.235018 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca" pod "42245ba5-9d23-479e-aee2-2e5d118c28bc" (UID: "42245ba5-9d23-479e-aee2-2e5d118c28bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:47.236609 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.236585 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs" pod "42245ba5-9d23-479e-aee2-2e5d118c28bc" (UID: "42245ba5-9d23-479e-aee2-2e5d118c28bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:47.236698 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.236642 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk" pod "42245ba5-9d23-479e-aee2-2e5d118c28bc" (UID: "42245ba5-9d23-479e-aee2-2e5d118c28bc"). InnerVolumeSpecName "kube-api-access-rs7mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:47.236763 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.236747 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls" pod "42245ba5-9d23-479e-aee2-2e5d118c28bc" (UID: "42245ba5-9d23-479e-aee2-2e5d118c28bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:47.335962 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.335932 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42245ba5-9d23-479e-aee2-2e5d118c28bc-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:47.335962 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.335958 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs7mk\" (UniqueName: \"kubernetes.io/projected/42245ba5-9d23-479e-aee2-2e5d118c28bc-kube-api-access-rs7mk\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:47.336167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.335970 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:47.336167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.335979 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42245ba5-9d23-479e-aee2-2e5d118c28bc-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:47.629011 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.628980 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:23:47.632890 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:47.632865 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c501563-568688dc5b-5msrd"] Oct 30 22:23:48.628951 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:48.628913 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:48.952250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:48.952166 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:49.427588 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:49.427557 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42245ba5-9d23-479e-aee2-2e5d118c28bc" path="/var/lib/kubelet/pods/42245ba5-9d23-479e-aee2-2e5d118c28bc/volumes" Oct 30 22:23:53.516127 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:53.516096 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:53.774314 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:53.774241 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:55.134610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:55.134573 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:55.786468 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:55.785914 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:55.984670 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:55.984610 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:59.906909 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:23:59.906871 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:00.110121 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:00.110069 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:03.224505 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:03.224461 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:03.415487 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:03.415434 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.249308 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.249273 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:24:06.249774 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.249759 4918 cpu_manager.go:414] "containerMap: removing container" podUID="42245ba5-9d23-479e-aee2-2e5d118c28bc" containerName="maas-api" Oct 30 22:24:06.249827 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.249774 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42245ba5-9d23-479e-aee2-2e5d118c28bc" containerName="maas-api" Oct 30 22:24:06.249903 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.249894 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="42245ba5-9d23-479e-aee2-2e5d118c28bc" containerName="maas-api" Oct 30 22:24:06.254570 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.254551 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.302138 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.302113 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.302245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.302142 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.302245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.302171 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpkw\" (UniqueName: \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.302389 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.302368 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.303700 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.303680 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8123f1-serving-cert\"" Oct 30 22:24:06.354242 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.354218 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:24:06.403149 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.403118 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.403315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.403159 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.403315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.403179 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.403315 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.403197 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpkw\" (UniqueName: \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.403315 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:24:06.403292 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-serving-cert: secret "maas-api-e2e-authcfg-http-8123f1-serving-cert" not found Oct 30 22:24:06.403549 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:24:06.403401 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls podName:9126f99e-ea7b-4bcf-be08-73a480863763 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:06.903377167 +0000 UTC m=+4227.596859395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls") pod "maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" (UID: "9126f99e-ea7b-4bcf-be08-73a480863763") : secret "maas-api-e2e-authcfg-http-8123f1-serving-cert" not found Oct 30 22:24:06.403950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.403925 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.405538 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.405519 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.463180 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.463156 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpkw\" (UniqueName: \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.482388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.482367 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.694650 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.694620 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.891186 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.891153 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.908541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.908512 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:06.910878 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:06.910851 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:07.167815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:07.167786 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:07.536921 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:07.536890 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:24:07.538316 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:24:07.538282 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9126f99e_ea7b_4bcf_be08_73a480863763.slice/crio-4bb2813c057bcda12a38dd6f5412279f90526f3cbca57171c7c0b1929bc83888 WatchSource:0}: Error finding container 4bb2813c057bcda12a38dd6f5412279f90526f3cbca57171c7c0b1929bc83888: Status 404 returned error can't find the container with id 4bb2813c057bcda12a38dd6f5412279f90526f3cbca57171c7c0b1929bc83888 Oct 30 22:24:08.346892 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:08.346852 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" event={"ID":"9126f99e-ea7b-4bcf-be08-73a480863763","Type":"ContainerStarted","Data":"4bb2813c057bcda12a38dd6f5412279f90526f3cbca57171c7c0b1929bc83888"} Oct 30 22:24:09.355719 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:09.355672 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" event={"ID":"9126f99e-ea7b-4bcf-be08-73a480863763","Type":"ContainerStarted","Data":"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf"} Oct 30 22:24:09.356158 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:09.355870 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:09.471472 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:09.471412 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" podStartSLOduration=2.549909794 podStartE2EDuration="3.471393033s" totalImagesPullingTime="921.483239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:09.467052022 +0000 UTC m=+4230.160534259" watchObservedRunningTime="2026-10-30 22:24:09.471393033 +0000 UTC m=+4230.164875270" Oct 30 22:24:11.579208 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:11.579167 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:15.366842 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:15.366816 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:24:15.805293 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:24:15.805201 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:24:16.796492 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:16.796454 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:21.848858 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:21.848792 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:24.901126 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:24.901079 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:27.086618 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:27.086582 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:32.062781 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:32.062745 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:33.878319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:33.878282 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:37.074981 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:37.074945 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:42.083790 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:42.083756 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:44.661974 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:44.661943 4918 scope.go:122] "RemoveContainer" containerID="0adbdeb16d49d1b4e9737da9cd7b22353cabe4c8011a51b15f1ea450eb227a90" Oct 30 22:24:45.800782 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:24:45.800742 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:24:47.098841 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:47.098797 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:51.986076 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:51.986038 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:53.914402 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:53.914365 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:54.314576 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:54.314540 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:56.998879 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:24:56.998846 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:02.294649 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:02.294612 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:07.402059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:07.402026 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:12.296999 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:12.296941 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:15.802008 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:25:15.801975 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:25:17.275291 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:17.275251 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:22.179916 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:22.179873 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:26.355848 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:26.355810 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:27.178798 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:27.178753 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:32.129929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:32.129896 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:37.089033 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:37.089000 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:42.069925 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:42.069892 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:45.804141 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:25:45.804109 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:25:47.200541 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:47.200479 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:52.240941 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:52.240906 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:54.205884 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:54.205852 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:54.507170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:54.507078 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:57.601431 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:57.601397 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:57.804281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:25:57.804241 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:01.238151 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:01.238116 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:06.333556 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.333517 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:26:06.333986 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.333741 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" podUID="9126f99e-ea7b-4bcf-be08-73a480863763" containerName="maas-api" containerID="cri-o://373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf" gracePeriod=30 Oct 30 22:26:06.479349 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.479317 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:06.592217 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.592160 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:26:06.675503 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.675476 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") pod \"9126f99e-ea7b-4bcf-be08-73a480863763\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " Oct 30 22:26:06.675650 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.675541 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\") pod \"9126f99e-ea7b-4bcf-be08-73a480863763\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " Oct 30 22:26:06.675700 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.675674 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\" (UniqueName: \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\") pod \"9126f99e-ea7b-4bcf-be08-73a480863763\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " Oct 30 22:26:06.675793 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.675753 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\") pod \"9126f99e-ea7b-4bcf-be08-73a480863763\" (UID: \"9126f99e-ea7b-4bcf-be08-73a480863763\") " Oct 30 22:26:06.675994 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.675969 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca" pod "9126f99e-ea7b-4bcf-be08-73a480863763" (UID: "9126f99e-ea7b-4bcf-be08-73a480863763"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:06.676167 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.676147 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9126f99e-ea7b-4bcf-be08-73a480863763-openshift-service-ca\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:06.677605 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.677574 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls" pod "9126f99e-ea7b-4bcf-be08-73a480863763" (UID: "9126f99e-ea7b-4bcf-be08-73a480863763"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:06.677742 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.677677 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs" pod "9126f99e-ea7b-4bcf-be08-73a480863763" (UID: "9126f99e-ea7b-4bcf-be08-73a480863763"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:06.677821 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.677761 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw" pod "9126f99e-ea7b-4bcf-be08-73a480863763" (UID: "9126f99e-ea7b-4bcf-be08-73a480863763"). InnerVolumeSpecName "kube-api-access-wjpkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:06.777286 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.777259 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpkw\" (UniqueName: \"kubernetes.io/projected/9126f99e-ea7b-4bcf-be08-73a480863763-kube-api-access-wjpkw\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:06.777286 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.777285 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-metrics-certs\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:06.777448 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:06.777296 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9126f99e-ea7b-4bcf-be08-73a480863763-maas-api-tls\") on node \"ip-10-0-132-23.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:07.050184 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.050143 4918 generic.go:356] "Generic (PLEG): container finished" podID="9126f99e-ea7b-4bcf-be08-73a480863763" containerID="373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf" exitCode=0 Oct 30 22:26:07.050438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.050204 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" Oct 30 22:26:07.050438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.050236 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" event={"ID":"9126f99e-ea7b-4bcf-be08-73a480863763","Type":"ContainerDied","Data":"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf"} Oct 30 22:26:07.050438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.050302 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz" event={"ID":"9126f99e-ea7b-4bcf-be08-73a480863763","Type":"ContainerDied","Data":"4bb2813c057bcda12a38dd6f5412279f90526f3cbca57171c7c0b1929bc83888"} Oct 30 22:26:07.050438 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.050324 4918 scope.go:122] "RemoveContainer" containerID="373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf" Oct 30 22:26:07.061370 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.061352 4918 scope.go:122] "RemoveContainer" containerID="373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf" Oct 30 22:26:07.061640 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:26:07.061617 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf\": container with ID starting with 373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf not found: ID does not exist" containerID="373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf" Oct 30 22:26:07.061758 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.061643 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf"} err="failed to get container status \"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf\": rpc error: code = NotFound desc = could not find container \"373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf\": container with ID starting with 373c9346ce6598a1c877c1e4d3ca69f3d026356ecb024dc0f7e56a833e3847cf not found: ID does not exist" Oct 30 22:26:07.206625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.206592 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:26:07.211026 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.210998 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8123f1-5cf6c9b7d5-lqhgz"] Oct 30 22:26:07.427479 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:07.427450 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9126f99e-ea7b-4bcf-be08-73a480863763" path="/var/lib/kubelet/pods/9126f99e-ea7b-4bcf-be08-73a480863763/volumes" Oct 30 22:26:08.038771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:08.038740 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:08.393747 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:08.393698 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:09.212041 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:09.212001 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:15.803219 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:26:15.803183 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:26:15.928629 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:15.928598 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:16.145780 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:16.145736 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:16.343078 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:16.343042 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:21.050821 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:21.050739 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh"] Oct 30 22:26:21.097993 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:21.097963 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7mvvh"] Oct 30 22:26:21.427245 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:21.427211 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c009168-e40e-4001-8add-7e3f01f448cc" path="/var/lib/kubelet/pods/3c009168-e40e-4001-8add-7e3f01f448cc/volumes" Oct 30 22:26:45.804582 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:26:45.804537 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:26:54.558612 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:54.558582 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:57.049749 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:57.049701 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc"] Oct 30 22:26:57.097904 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:57.097864 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fwtwc"] Oct 30 22:26:57.426929 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:26:57.426887 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63b9089-1f47-41e4-a0c4-5c381e17e050" path="/var/lib/kubelet/pods/d63b9089-1f47-41e4-a0c4-5c381e17e050/volumes" Oct 30 22:27:03.425086 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:03.425040 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:03.625740 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:03.625140 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:15.803722 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:27:15.803676 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:27:28.241399 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:28.241361 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:28.459794 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:28.459687 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:29.852391 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:29.852343 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hpcg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:29.852946 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:29.852637 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hpcg6" podUID="e33be08a-8641-49fd-9b8e-87231d6d10e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:30.060367 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:30.060330 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4"] Oct 30 22:27:30.106747 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:30.106626 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00985ecb-52cc4"] Oct 30 22:27:31.427607 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:31.427567 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6b0769-f882-45f9-8388-4ccd60cdc65a" path="/var/lib/kubelet/pods/7a6b0769-f882-45f9-8388-4ccd60cdc65a/volumes" Oct 30 22:27:39.023722 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:39.023677 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wr4p8_9014e08f-c7de-43db-a003-f5513fc49878/manager/0.log" Oct 30 22:27:39.261656 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:39.261631 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lgj9s_12243256-966e-46a5-94e7-f7190dd53c15/manager/0.log" Oct 30 22:27:40.876757 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:40.876729 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5cxn_6d0b4c70-9500-407d-beb3-e7044ca16689/manager/0.log" Oct 30 22:27:41.181963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:41.181892 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rvzxp_27fd99b3-1c94-498b-86f9-7c01e5b8576c/proxy/0.log" Oct 30 22:27:42.215876 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:42.215844 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6687d569c9-scd7k_24911709-2140-48d5-9a31-8159c398ebf5/maas-api/0.log" Oct 30 22:27:43.121241 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:43.121209 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4kd2z_672b478c-ed93-4fce-921e-70ea550d0b20/postgres/0.log" Oct 30 22:27:44.718156 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:44.718124 4918 scope.go:122] "RemoveContainer" containerID="8071c37108735848db425a0e5c22c8d2cc110d642b6fd2257b98070d63d65bbe" Oct 30 22:27:45.114368 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:45.114335 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b4hzg_8629159b-c31f-46e6-a8b4-ec64fbda1493/manager/0.log" Oct 30 22:27:45.341610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:45.341577 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7h4gh_3e114750-10c3-400d-92c9-051f0a56cde9/manager/0.log" Oct 30 22:27:45.801203 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:27:45.801169 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:27:45.835337 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:45.835310 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-65rf5_4bbe384e-e4d3-4917-9985-7fbb079f5e04/manager/0.log" Oct 30 22:27:46.427229 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:46.427198 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t252l_7e2402e4-cd7c-4715-b4f0-1a2ab78840f5/manager/0.log" Oct 30 22:27:47.280876 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:47.280846 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb_9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8/istio-proxy/0.log" Oct 30 22:27:48.909440 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:48.909402 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hpcg6_e33be08a-8641-49fd-9b8e-87231d6d10e1/istio-proxy/0.log" Oct 30 22:27:49.641309 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:49.641278 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-qzk86_b85f2df0-2e73-44aa-9f4f-caf9bec10c71/istio-proxy/0.log" Oct 30 22:27:50.103745 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:50.103646 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-smwqx_aef085da-a319-4c09-a236-af0467dc72ec/istio-proxy/1.log" Oct 30 22:27:50.911205 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:50.911178 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6_f2aaa457-d02c-4c2e-9db0-c5769374a3b2/istio-proxy/0.log" Oct 30 22:27:51.136546 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:51.136515 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xplfs_b8d73838-6d45-480e-8964-eba9637c8a09/payload-pre-processing/0.log" Oct 30 22:27:51.366345 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:51.366323 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:27:51.598121 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:51.598091 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b999d9c47-t55lr_94567a77-bf25-464b-adde-cfb50ccfea99/router/0.log" Oct 30 22:27:52.114185 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:52.114159 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-smwqx_aef085da-a319-4c09-a236-af0467dc72ec/istio-proxy/0.log" Oct 30 22:27:52.720796 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:52.720767 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7_b58abe86-5d5d-4384-bab5-23c330ced8a9/main/0.log" Oct 30 22:27:52.960279 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:52.960253 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-nwgzt_b93e5471-ae00-484f-a489-d1670b236686/main/0.log" Oct 30 22:27:53.204111 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:53.204088 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6gksg_95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9/main/0.log" Oct 30 22:27:53.910982 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:53.910953 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bjd7j_06a6362b-47ea-4e84-8bee-0a50e3643ac3/main/0.log" Oct 30 22:27:54.355430 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:54.355348 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt_8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad/main/0.log" Oct 30 22:27:54.899337 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:54.899305 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:55.374310 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:27:55.374284 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:05.239674 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:05.239640 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wr4p8_9014e08f-c7de-43db-a003-f5513fc49878/manager/0.log" Oct 30 22:28:05.507281 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:05.507207 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lgj9s_12243256-966e-46a5-94e7-f7190dd53c15/manager/0.log" Oct 30 22:28:06.925690 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:06.925659 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5cxn_6d0b4c70-9500-407d-beb3-e7044ca16689/manager/0.log" Oct 30 22:28:07.251251 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:07.251177 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rvzxp_27fd99b3-1c94-498b-86f9-7c01e5b8576c/proxy/0.log" Oct 30 22:28:08.134869 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:08.134841 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6687d569c9-scd7k_24911709-2140-48d5-9a31-8159c398ebf5/maas-api/0.log" Oct 30 22:28:09.146363 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:09.146317 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4kd2z_672b478c-ed93-4fce-921e-70ea550d0b20/postgres/0.log" Oct 30 22:28:10.892297 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:10.892266 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b4hzg_8629159b-c31f-46e6-a8b4-ec64fbda1493/manager/0.log" Oct 30 22:28:11.195170 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:11.195098 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7h4gh_3e114750-10c3-400d-92c9-051f0a56cde9/manager/0.log" Oct 30 22:28:11.697581 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:11.697551 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-65rf5_4bbe384e-e4d3-4917-9985-7fbb079f5e04/manager/0.log" Oct 30 22:28:12.392548 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:12.392506 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t252l_7e2402e4-cd7c-4715-b4f0-1a2ab78840f5/manager/0.log" Oct 30 22:28:13.057611 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:13.057578 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb_9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8/istio-proxy/0.log" Oct 30 22:28:14.848791 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:14.848766 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hpcg6_e33be08a-8641-49fd-9b8e-87231d6d10e1/istio-proxy/0.log" Oct 30 22:28:15.795365 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:15.795335 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-qzk86_b85f2df0-2e73-44aa-9f4f-caf9bec10c71/istio-proxy/0.log" Oct 30 22:28:15.805743 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:28:15.805688 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:28:16.397663 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:16.397630 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-smwqx_aef085da-a319-4c09-a236-af0467dc72ec/istio-proxy/1.log" Oct 30 22:28:17.293963 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:17.293933 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-7xfh6_f2aaa457-d02c-4c2e-9db0-c5769374a3b2/istio-proxy/0.log" Oct 30 22:28:17.554124 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:17.554048 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xplfs_b8d73838-6d45-480e-8964-eba9637c8a09/payload-pre-processing/0.log" Oct 30 22:28:17.813887 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:17.813806 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:28:18.066206 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:18.066126 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b999d9c47-t55lr_94567a77-bf25-464b-adde-cfb50ccfea99/router/0.log" Oct 30 22:28:18.712256 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:18.712228 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vg9c7_b58abe86-5d5d-4384-bab5-23c330ced8a9/main/0.log" Oct 30 22:28:18.958996 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:18.958946 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-nwgzt_b93e5471-ae00-484f-a489-d1670b236686/main/0.log" Oct 30 22:28:19.225225 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:19.225199 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6gksg_95e1109d-e3ce-4f99-a604-7ee8c3bb4fa9/main/0.log" Oct 30 22:28:20.053065 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:20.053032 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bjd7j_06a6362b-47ea-4e84-8bee-0a50e3643ac3/main/0.log" Oct 30 22:28:20.580610 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:20.580582 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-vx5kt_8a5e4ee2-bb87-4724-a9ea-b30656c8c6ad/main/0.log" Oct 30 22:28:40.149719 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.149675 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:28:40.163351 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.163321 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:28:40.165577 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.165552 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5zc7_735d7add-e2e7-4f2a-be19-2585f3bc731c/extract/0.log" Oct 30 22:28:40.168388 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.168366 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:28:40.178535 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.178517 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-crfvx_de5cba4a-8119-475d-a365-f7977405ea6e/console-operator/3.log" Oct 30 22:28:40.183461 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:40.183444 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gdshh_3b7a69c4-438f-4679-9255-d0cae7ffc19f/ovn-acl-logging/0.log" Oct 30 22:28:44.741206 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:44.741177 4918 scope.go:122] "RemoveContainer" containerID="6e436f416b7ad8729425447a6087aa4100a0f9b6e5ea9077dd6e6c8245f9d2fc" Oct 30 22:28:44.750536 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:44.750516 4918 scope.go:122] "RemoveContainer" containerID="ba0518c014e7db45df8ef0ef3989e7179b99505a78809756f6cbfddae314574e" Oct 30 22:28:45.800640 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:28:45.800604 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:28:51.213748 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:51.213721 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:28:51.440764 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:51.440731 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:28:55.288408 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:55.288370 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:55.527059 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:28:55.527023 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:10.946457 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:10.946422 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:15.801448 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:29:15.801411 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:29:18.352447 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:18.352415 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:18.543981 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:18.543945 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:32.483914 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:32.483845 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:29:32.711090 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:32.711058 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gnxv2_b41233f0-deeb-4c61-b54f-e5334219b9a5/payload-processing/0.log" Oct 30 22:29:33.076114 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.076082 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgcw8/must-gather-75lph"] Oct 30 22:29:33.076622 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.076606 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9126f99e-ea7b-4bcf-be08-73a480863763" containerName="maas-api" Oct 30 22:29:33.076671 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.076624 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9126f99e-ea7b-4bcf-be08-73a480863763" containerName="maas-api" Oct 30 22:29:33.076744 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.076733 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9126f99e-ea7b-4bcf-be08-73a480863763" containerName="maas-api" Oct 30 22:29:33.080250 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.080231 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.130198 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.130173 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgcw8\"/\"default-dockercfg-sqfbc\"" Oct 30 22:29:33.130880 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.130861 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:33.176820 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.176798 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgcw8\"/\"kube-root-ca.crt\"" Oct 30 22:29:33.176938 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.176822 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgcw8\"/\"openshift-service-ca.crt\"" Oct 30 22:29:33.179771 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.179747 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/must-gather-75lph"] Oct 30 22:29:33.231406 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.231379 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpsm\" (UniqueName: \"kubernetes.io/projected/1cb71195-3247-40a0-8dcf-00449d344c7c-kube-api-access-7hpsm\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.231518 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.231421 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1cb71195-3247-40a0-8dcf-00449d344c7c-must-gather-output\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.329202 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.329128 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:33.332213 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.332194 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1cb71195-3247-40a0-8dcf-00449d344c7c-must-gather-output\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.332320 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.332303 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpsm\" (UniqueName: \"kubernetes.io/projected/1cb71195-3247-40a0-8dcf-00449d344c7c-kube-api-access-7hpsm\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.332561 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.332539 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1cb71195-3247-40a0-8dcf-00449d344c7c-must-gather-output\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.393806 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.393771 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpsm\" (UniqueName: \"kubernetes.io/projected/1cb71195-3247-40a0-8dcf-00449d344c7c-kube-api-access-7hpsm\") pod \"must-gather-75lph\" (UID: \"1cb71195-3247-40a0-8dcf-00449d344c7c\") " pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.394586 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.394570 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/must-gather-75lph" Oct 30 22:29:33.771277 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.771241 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/must-gather-75lph"] Oct 30 22:29:33.772910 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:29:33.772883 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cb71195_3247_40a0_8dcf_00449d344c7c.slice/crio-3c5ef83667023ad0a97f0fd56bb2c7e30b4055b6b932b6766e4b7b8f9a15339f WatchSource:0}: Error finding container 3c5ef83667023ad0a97f0fd56bb2c7e30b4055b6b932b6766e4b7b8f9a15339f: Status 404 returned error can't find the container with id 3c5ef83667023ad0a97f0fd56bb2c7e30b4055b6b932b6766e4b7b8f9a15339f Oct 30 22:29:33.774816 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:33.774798 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:29:34.380761 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:34.380728 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/must-gather-75lph" event={"ID":"1cb71195-3247-40a0-8dcf-00449d344c7c","Type":"ContainerStarted","Data":"3c5ef83667023ad0a97f0fd56bb2c7e30b4055b6b932b6766e4b7b8f9a15339f"} Oct 30 22:29:35.389157 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:35.389126 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/must-gather-75lph" event={"ID":"1cb71195-3247-40a0-8dcf-00449d344c7c","Type":"ContainerStarted","Data":"c55352dc7792388721508cbe555c31a37a17ec352e815c307d97701ef219ff62"} Oct 30 22:29:36.406206 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:36.406169 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/must-gather-75lph" event={"ID":"1cb71195-3247-40a0-8dcf-00449d344c7c","Type":"ContainerStarted","Data":"2ff296bd25645c96ebec76fc20a548173375e385b7a84c544383072b80382aa7"} Oct 30 22:29:36.517035 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:36.516978 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgcw8/must-gather-75lph" podStartSLOduration=2.12181402 podStartE2EDuration="3.516962624s" totalImagesPullingTime="1.395148604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:36.515863676 +0000 UTC m=+4557.209345911" watchObservedRunningTime="2026-10-30 22:29:36.516962624 +0000 UTC m=+4557.210444861" Oct 30 22:29:44.261729 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:44.261344 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-47nbz_d7cd2ff9-7aeb-4ebf-9605-a213958c1153/global-pull-secret-syncer/0.log" Oct 30 22:29:45.544730 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:45.544088 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c7qqx_88f63d46-40a6-46c0-b359-d58509ccba89/connection-checker/0.log" Oct 30 22:29:45.827040 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:29:45.826946 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor" Oct 30 22:29:45.968762 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:45.968730 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jz9n2_323fa83f-8f63-4a65-9837-d31e6f6de11d/connection-checker/0.log" Oct 30 22:29:46.408731 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:46.405667 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-m4s2v_f9771a55-0c02-4b30-9fe7-1850bd414cde/connection-checker/0.log" Oct 30 22:29:46.860950 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:46.860919 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" Oct 30 22:29:48.748691 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:48.748651 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-23.us-west-2.compute.internal_76d614578e9abb3302e5026d743d87fe/haproxy/0.log" Oct 30 22:29:51.505675 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:51.505643 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b4hzg_8629159b-c31f-46e6-a8b4-ec64fbda1493/manager/0.log" Oct 30 22:29:51.920065 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:51.919992 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-7h4gh_3e114750-10c3-400d-92c9-051f0a56cde9/manager/0.log" Oct 30 22:29:52.703453 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.703409 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt"] Oct 30 22:29:52.711815 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.711783 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.780000 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.779970 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-65rf5_4bbe384e-e4d3-4917-9985-7fbb079f5e04/manager/0.log" Oct 30 22:29:52.795143 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.795108 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt"] Oct 30 22:29:52.855920 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.855881 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhh4\" (UniqueName: \"kubernetes.io/projected/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-kube-api-access-xwhh4\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.856117 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.856028 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-podres\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.856117 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.856073 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-sys\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.856254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.856159 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-lib-modules\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.856254 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.856193 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-proc\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957236 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-podres\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957319 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957287 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-sys\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957351 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-lib-modules\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957379 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-proc\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957392 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-podres\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957425 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhh4\" (UniqueName: \"kubernetes.io/projected/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-kube-api-access-xwhh4\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957438 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-sys\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957486 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-lib-modules\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:52.957553 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:52.957494 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-proc\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:53.015384 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.015356 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhh4\" (UniqueName: \"kubernetes.io/projected/e8522f7f-cd04-4d4d-862d-5e809c4f44e9-kube-api-access-xwhh4\") pod \"perf-node-gather-daemonset-7j6bt\" (UID: \"e8522f7f-cd04-4d4d-862d-5e809c4f44e9\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:53.026238 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.026211 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:53.402760 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.402731 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt"] Oct 30 22:29:53.405109 ip-10-0-132-23 kubenswrapper[4918]: W1030 22:29:53.405079 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8522f7f_cd04_4d4d_862d_5e809c4f44e9.slice/crio-76a5acc106ec621df5eeb1f4f8144b6066e503f58726c8272569b29742303f63 WatchSource:0}: Error finding container 76a5acc106ec621df5eeb1f4f8144b6066e503f58726c8272569b29742303f63: Status 404 returned error can't find the container with id 76a5acc106ec621df5eeb1f4f8144b6066e503f58726c8272569b29742303f63 Oct 30 22:29:53.603080 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.603045 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-t252l_7e2402e4-cd7c-4715-b4f0-1a2ab78840f5/manager/0.log" Oct 30 22:29:53.603236 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.603118 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" event={"ID":"e8522f7f-cd04-4d4d-862d-5e809c4f44e9","Type":"ContainerStarted","Data":"8dc4253c0c8718c3f1c179cb0e63d1ef46bef59e34a565245ed7c848dcdc2d5c"} Oct 30 22:29:53.603236 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.603171 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" event={"ID":"e8522f7f-cd04-4d4d-862d-5e809c4f44e9","Type":"ContainerStarted","Data":"76a5acc106ec621df5eeb1f4f8144b6066e503f58726c8272569b29742303f63"} Oct 30 22:29:53.603236 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.603194 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:29:53.716776 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:53.715077 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" podStartSLOduration=1.715057536 podStartE2EDuration="1.715057536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:53.714321191 +0000 UTC m=+4574.407803429" watchObservedRunningTime="2026-10-30 22:29:53.715057536 +0000 UTC m=+4574.408539774" Oct 30 22:29:55.707860 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:55.707829 4918 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5d68_2da5bf9f-3892-4b54-a337-880b1e19faf8/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:58.922691 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:58.922662 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-thvb5_e498e377-392b-421d-a455-1f7624968e6a/cluster-monitoring-operator/0.log" Oct 30 22:29:59.622625 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:29:59.622597 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-7j6bt" Oct 30 22:30:02.647361 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:30:02.647329 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghxxm_33b37892-bcc7-422f-a15d-71b0abeed209/node-exporter/0.log" Oct 30 22:30:03.046260 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:30:03.046180 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghxxm_33b37892-bcc7-422f-a15d-71b0abeed209/kube-rbac-proxy/0.log" Oct 30 22:30:03.443702 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:30:03.443669 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghxxm_33b37892-bcc7-422f-a15d-71b0abeed209/init-textfile/0.log" Oct 30 22:30:15.303488 ip-10-0-132-23 kubenswrapper[4918]: I1030 22:30:15.303461 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccnt7mb_9d8ce2d4-02c2-448c-ad3c-337a0e4bc7e8/istio-proxy/0.log" Oct 30 22:30:15.807462 ip-10-0-132-23 kubenswrapper[4918]: E1030 22:30:15.807425 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bn5sg" containerName="extractor"